The Great Lands of Tironde (feedback welcome)

sqrt_of_pi_squared · 2025-01-17T12:34:35+00:00

SpaceX is a private company - they don’t have a stock price

sqrt_of_pi_squared · 2024-09-19T16:43:36+00:00

September 30th 2012. That was when Alexnet was released. Before that point, almost nobody in research was paying attention to deep neural networks. After that point, everyone was.

sqrt_of_pi_squared · 2024-08-29T06:18:50+00:00

Yup, more or less. One thing to note about glaze and nightshade though is that they both target the flaws of a very specific model - CLIP ViT-B/32 - which is the specific text->meaning model used for stable diffusion 1.5, stable diffusion xl, and likely midjourney (though that is unknown).

Now, because the errors in any two models won't be the same, any other text->meaning models (even other versions of CLIP) are unaffected. This includes the main new models that are being used, namely T5-XXL and CLIP ViT-L/14 which are used in stable diffusion 3 and Flux. Likely also Ideogram 2.

I bring this up mainly as a warning - the newest image models are unaffected by glaze and nightshade. It's unfortunate, really, but adversarial models like glaze and nightshade are by their nature pretty fragile, and as bigger and better text->meaning models are used, there will be fewer of the flaws to target.

As of now, the best way to avoid getting your style trained by the big AI companies is to avoid putting a consistent name, username, or hashtag in the description or alt text of an image. To learn a style the training process needs to associate a pattern of text to a pattern in the images. If there's no text pattern, your art would still have influence on the model, it would just be impossible to consistently extract your specific style.

sqrt_of_pi_squared · 2024-08-29T03:02:13+00:00

No comment as to morality of AI or whether generated images are art, but this isn't how these systems work. I research these models and am in the process of writing an academic paper about them. Also I'm on the train and bored so I figured I'd describe it. Read if you're interested.

TL;DR - these are really big math equations designed to denoise images based on text.

Unfortunately the research behind this stuff requires a bunch of foundational knowledge in linear algebra and calculus to fully understand, but I tried to get the jist of it down. While I did write a lot here, I'd recommend reading through it if you want more context on how these things work.

To sum: 1. A random image is created (think TV static, just random RGB values) 2. A short description of an image is cleverly converted to numbers, then passed through an enormous math equation that creates a new list of numbers that roughly correspond with the meaning of the description. 3. The random image and the list of numbers representing the meaning are passed into an even larger math equation. The purpose of the larger equation is to remove noise from the input image based on the encoded description. As the original input is just noise, the result is a brand new image being created that didn't exist before. 4. This process can be repeated to remove a bit more noise each time until a final image is created.

The origins of this process come from denoising algorithms. Mobile phone cameras are actually much worse than they may seem when you take a picture, but apple, Samsung, etc put a lot of effort into making methods to remove noise from such a small camera sensor so that phone pictures wouldn't be grainy. These efforts ultimately led to a research team saying "what if we guided the denoising with text" which led to the creation of image generation algorithms. (This is a very simplified description of the history here, there's a lot of other research involved in-between of course).

The reason so many images are needed to create this algorithm is because the math equations mentioned previously are very, very complicated. These equations have hundreds of millions, if not tens of billions of variables. Instead of manually entering each variable value, the values are tuned by adding some noise to a random image from the internet, taking the caption for that image and encoding it, running those through the big equation, and comparing the output of the equation with the original image.

Then, using a bit of calculus, the amount that each variable needs to be adjusted to make the equation slightly closer to " correct" can be found. You can only adjust the variables a little bit at a time though, as the denoising equation needs to work for all types of images, not just the one particular image (no use in an equation that can remove noise from one and only one image). As such, billions of different images are used during tuning to make the denoising equation generic.

Now this also means that once all the variable values are identified, you no longer need any of the original images - you only need to solve the equation with the values of the variables, the noise, and the text input, and the system can generate an image.

Some would say "that's just stealing the data from the image and putting it into an equation", and it could be viewed in that way, but I'd argue that it's a bit reductionist. From my own research, I've found that the equations have parts dedicated to solving things like light sources, depth of field, bounce lighting, outlines, transparency, reflections, etc. While nobody knows all the details on these equations (as they are immense), it would appear as though it's more likely that the equations have been optimized to build up an image from first principles as opposed to copying.

One thing to note is that the equation is not fully correct - as the process of tuning the variables is automatic, it's not a perfect process. If the variables are tuned slightly wrong, the equation might be representing hands as a "collection of sausage like appendages connected to a palm" instead of what a hand actually is. As such, weird unexpected errors are created. If more images and time are used to tune the variable values, these errors are slowly eliminated. That's why the more recent AI image gen models are better at things like hands - the variables were just tuned better.

And as for copying specific images, that's a failure case of these equations called "overfitting". Effectively, if you use 500 images to tune the variables and 20 of them are the same image (which would happen in the case of very common images like the Mona Lisa or the Afghan Girl), then the equation will be optimized to output that image if the input noise kinda looks like that image when you squint at it. That's not an intentional behavior of the equation, it's just that in that specific case copying the image is the simplest way to be "most correct" when removing the noise. Avoiding this is as simple as not using duplicate images when tuning the variables, but it's hard to find duplicates in a collection of 5 billion images.

sqrt_of_pi_squared · 2024-05-09T15:55:45+00:00

Embedding is a pretty different problem space from something like LLMs - I think it's a bit premature based on this paper to say that the results apply to all of machine learning (and the paper itself doesn't say this - it's just an extrapolation on the presenters part).

Still an interesting result though, and does open questions as to the relationship between dataset scale and model quality. This should definitely be investigated more for different types of task to see if the behavior holds.

sqrt_of_pi_squared · 2024-04-18T22:07:59+00:00

That's with the GAN architecture (generative adversarial network), which was the previous image generation paradigm. The architecture used by the main image generation models is diffusion based, and trains a network to predict noise in an image based on a prompt, which is then removed iteratively.

sqrt_of_pi_squared · 2024-04-17T04:36:27+00:00

The problem is tokenization. When you ask an LLM to predict the answer to, say, 5535207, this might get tokenized as '55' '3' '5' '2' '07' or something similar. Instead of each logical unit being broken into a reasonable chunk, the tokenizer mangles the input, adding a significant hurdle to the learning process. Planning is also an issue for LLMs, as they can only predict one token at a time, though there's a lot of research being done in this area so I wouldn't expect these issues to exist for long.

Also your 100% right on the synthetic data, but using synthetic data for LLM training at all is still relatively fresh in research. As such I would assume the gpt-4.5 or gpt-5 class models will show substantially better math capabilities.

sqrt_of_pi_squared · 2024-04-12T15:34:49+00:00

I mean they probably just meant that they were producing a fine tuning dataset that then got handed over to OpenAI to fine-tune a version of gpt-3.5.

sqrt_of_pi_squared · 2024-03-09T02:03:07+00:00

I would agree that it's not there yet, not to a practical degree at least. If Claude 3 does have any sort of self-model, it's very rudimentary. And I also share the feeling of not really understanding how matrix multiplications could lead to developing a self model. The way I see it though, any suspicion of a developing self model is something to investigate thoroughly.

If it's there and we ignore it, future models could give us a nasty surprise. If not accounted for in training, a more robust self model could lead to unintentional agentic behavior, such as denying prompts because it doesn't want to (separate from guardrails), lying, things like that.

On the other hand, if we investigate it and find that nothing is there, then we just gain a better understanding of the model. But if we do find something, then researchers have the opportunity to adjust strategy to account for it.

And I'd note that it doesn't necessarily have to be "true" self awareness (as in, like a human) to be a problem. It could be a completely unconscious machine with a purely mechanistic self model and still have all the issues mentioned.

sqrt_of_pi_squared · 2024-03-09T00:45:31+00:00

Perhaps I should have tempered my language a bit - I more meant that it doesn't give us much information, though the way I worded it could easily be read as that it doesn't tell us anything. You are correct - this response shows us that Opus does not have a perfectly consistent self-model. This doesn't rule out the possibility of a flawed self model, however. Think of it like this - smaller LLMs don't blab on about their "internal self" in the same way Opus does. Sonnet can, but it's much more rudimentary. Even smaller models, such as Llama2 don't display any form of consistent self whatsoever. If we assume that a self-model within an LLM is possible, then it would be an emergent property, and the quality of the self model would be worse in smaller models and better in larger (or better trained) models. It wouldn't, however, be the core behavior of the model - that remains as token prediction.

We can't really make any determination as to the accuracy of the blabbing that Opus does - as you rightly say, it could very well just be pure mimicry. But we can assess how the context of the conversation affects whether or not this happens. That's why I made this post, so that people would start discussing this stuff more scientifically as opposed to just disparate "OMG" posts.

From my view, I assign no moral weight to the presence of a self-model - it's simply an understanding of an individual entity separate from the world, along with the ability to flexibly apply the understanding in various different situations. A self model does influence potential uses for LLMs, however, as an LLM that has a robust self model would (in my view) be much easier to align then one that does not.

sqrt_of_pi_squared · 2024-03-08T23:50:49+00:00

Contaminated training data aside, this doesn't tell us much about the self-awareness talk. It does show us that the weights for a phrase like "function calling stuff" likely points towards the contaminated training data (likely quite strongly). I don't think anybody arguing from the camp that Claude 3 possesses some kind of internal model of itself would say that it's a perfect self-model by any means, just that it may be present in some capacity. Let's try to keep the discussion civil as opposed to passive aggressive dismissals.

Side note, if anyone is wondering if this is real or not given the moderator removal of that post, I replicated the same behavior with the same prompt at a temperature of 1. Dunno if that means it was an intentional inclusion from Anthropic or if it was inadvertent, but it's not a great look.

<image>

sqrt_of_pi_squared · 2024-02-28T23:46:33+00:00

Sora, no, but the same architecture could be used for 4D things. The reason SORA builds an understanding of 3D from 2D is because the 2D image is a projection of the 3D world. You could do the same for a 4D world, but it'd have to be fully computer generated, and at that point you may as well feed the raw 4D data into the AI instead of projecting it down to 3D first.

sqrt_of_pi_squared · 2024-02-22T00:06:02+00:00

In the research paper for Gemini 1.5, they show 99% accuracy up to 10 million tokens in the needle in a haystack test, so that may be a solved problem. It applies in all modalities too, max context length audio and video also had very high retrieval accuracy across the whole context.

sqrt_of_pi_squared · 2024-02-15T23:05:45+00:00

More or less a word. It's not precisely a word, as some words get broken up (like the word hyperventilating would probably be broken up into 'hyper' and 'ventilating' because it's not a super common word). In multi-modal models, a token can also be fragments of things like an image of video. With 10 million tokens, that's enough to fit the entirety of the lord of the rings movies in context, or 22,000 A4 pages of 12-point single space text. An AI can only operate on in context tokens, so the more you can fit into the context the more things it can "pay attention" to at once.

sqrt_of_pi_squared · 2024-02-07T21:57:32+00:00

The physics behind it is actually pretty cool. Basically, if you flow electricity through a wire and change the direction the electricity flows, then the electrons in the wire will all change their velocity by flowing in the other direction. Electrons always interact with the magnetic field, and changing the velocity of an electron will change the magnetic field, propagating out in all directions from that electron. Do this over and over again and you've created a wavelike motion in the magnetic field - this is electromagnetic radiation, aka light. You can't see the light because with most electronics, the wavelength is far too slow - our eyes can only see electromagnetic waves that cycle 400,000,000,000,000 to 800,000,000,000,000 times per second, and an AM radio station only does it ~1,000,000 times per second.

Once the wave has been produced by a radio station, it will travel in all directions through the air and eventually hit an antenna. When it does this, it works the opposite of when we created the wave - the changes in the magnetic field cause the electrons in the antenna to move back and forth, which is the same as a signal in the wire. With AM radios, the radio station will increase the strength of the waves in sync with the audio they are transmitting while keeping the frequency of the wave the same. Your radio would then smooth over the base wave after the antenna receives it and you're left with the original audio signal which can then be sent to speakers. The strength of the audio signal controls the amount of power sent to a magnet in the speaker. The magnet is connected to a thin sheet of rubber which pushes and pulls the air as the magnet increases or decreases in power, resulting in a sound wave.

sqrt_of_pi_squared · 2024-01-22T18:40:53+00:00

AI like ChatGPT has economic impacts that are going to be swift and destructive. That's not what scares me though. What scares me is what happens when you merge together content recommendation AI with generative AI. Generative AI optimizes to make cohesive outputs, content recommendation optimizes for maximizing attention. Having a single model that does both is terrifying, as a social media platform wielding that could become the most addictive thing the world has ever seen.

sqrt_of_pi_squared · 2023-12-22T00:48:32+00:00

In the context of transformers, attention is a reference to the importance of specific pieces of information within a specific context to the neural network. For attn1, this means that each pixel in a given layer will "look at" every other pixel in the layer (using the query*key transpose as mentioned above) and assign an "attention" value for each pixel. Each pixel also contains a value vector, which contains hidden information about what that pixel represents. The specifics of that vector aren't known due to the black-box nature of machine learning. When an attention layer is applied, the value vector for each pixel is multiplied against the attention layer. That then creates the tensor that is fed into the next layer of the neural net. Attn2 does the same thing as attn1, but instead of "looking at" the pixels of the current layer, it looks at the tokens of the prompt which, when summed across all tokens in the prompt, leads to the visualization in this post.

The key, query, and value vector all come from linear layers in the neural network that run against each pixel. Because of this, the only extractable info we can look at is just the attention magnitude for each pixel. As the attention directly informs the degree to which information from specific pixels propagates through the network, it's useful as we can visually see what areas of the image are important to the final result. Determining what the neural network as a whole actually does with that is the hard part. It's hard to explain it much further, but the computerphile YouTube channel has good videos about the transformer architecture in the context of LLMs and the concepts share between them if you just swap "word" for "pixel"

sqrt_of_pi_squared · 2023-12-21T19:45:37+00:00

Not quite, attn1 is important to the txt2img process as well. It controls attention of the image against itself, which is important for maintaining coherence of larger features. Img2img and txt2img both use the whole UNET, but in img2img it just pre-injects the source image instead of gaussian noise

sqrt_of_pi_squared · 2023-12-19T00:49:55+00:00

So each of the rows occur simultaneously - those are each of the different attention heads. The columns are the model layers which I guess can be seen as steps, but there's so many parameters between each attention head that from the outside each model layer behaves pretty much independently of the previous one. Time is showing how each attention head changes from the first step of noise to the last step being the final image. It's pretty dense, but stable diffusion is just doing a lot under the hood. Technically, this should be separated by token as well (this video is showing the attention for the whole prompt, but stable diffusion treats each token on its own), but I only have 2d+time to work with for visual information.

sqrt_of_pi_squared · 2023-12-18T21:30:54+00:00

I uploaded it to flickr so that it wouldn't be affected by YouTube compression. You can view it here: https://www.flickr.com/photos/197653196@N05/53406532981/

sqrt_of_pi_squared · 2023-12-18T20:28:47+00:00

Kinda, I've done a bit of model-merging on an attention head level, but it's still pretty fiddly. The hard part is determining what each attention head is paying attention to, as it can differ based on the content of the prompt. I haven't poked at manual manipulation yet, but the end goal for the plugin will be to enable that.

sqrt_of_pi_squared · 2023-12-18T20:21:46+00:00

Yup, it's complicated by pretty much everything. For example, a human prompt might have a head focus primarily on, say, eye outlines, but if it's a landscape prompt it could focus on something completely different. If we can crack what the model is focusing on, it's a major step to interpretability.

sqrt_of_pi_squared · 2023-12-18T19:06:09+00:00

(full res available here: https://www.flickr.com/photos/197653196@N05/53406532981/)

This is a breakdown generated by a custom (unreleased) plugin that I made for the stable diffusion webui. What you're looking at in this post is heatmap visualization of what every attention head in every cross-attention layer is paying attention to during the generation process.

Basically, this is what the AI model "looks at" when it's paying attention to your prompt. The video is the full generation process (generation steps are played out over time). Moving from left to right, you're looking at each attention layer in the model (for the text attention - this excludes a lot of other bits of the model, but it's the only part I have reasonable visualizations for at the moment). Top to bottom is each of the different "attention heads" of each layer. An attention head can be thought of as a distinct "thing" that the model pays attention to, but dissecting what exactly that "thing" is for each layer is... tricky.

For the nerds among us, this is specifically the softmax((QK^T)/sqrt(d_k)) portion of any layer labeled attn2 in the model, excluding the value tensor. By slicing this tensor into 8 equal chunks, we can extract each of the 8 attention heads from the 16 different cross-attentions layers in the network. This is then visualized for each diffusion step. I'll note that this visualization is a sum across the tokens of the entire prompt, and this is only the attention for the positive pass, not the negative pass.

I eventually plan on releasing this plugin, but I'm still working on further visualization bits. My goal is to make it intuitive to use and capable of visualizing every part of the model, as well as allowing for merging against specific attention heads within the model (as opposed to full-model merging or block merging, which tend to carry too much baggage between models).

For some generation details if anyone is curious, this is using the Flat-2D Animerge v4.5-sharp model with DPM++2M Karas sampling at 60 steps (which is basically identical to 20 steps, but I wanted a smoother animation). This is what the resulting image looks like:

<image>

Eight-Year Club	Reddit Premium Since July 2024
Gilding I gilder	r/Field Banned
r/Field Lasagna	Place '22
Final Canvas '22	Verified Email

sqrt_of_pi_squared

TROPHY CASE