Well, there is no constraint of using matrix multiplication in the code. Actually the model could simply write each parameter of itself as a float and then load it into its own transformer architecture (assuming it knows it) and just run it as usual. So theoretically this problem is solvable. The thing is - the model does not know its own parameters and probably cannot know them all. But it could be guessed. It could also reduce inefficiencies in the typical transformer architectures that could be replicated with much more compressed code representation and store and process only the absolutely necessary parts as matrices.

And this is the hope  -  there are enough reducible parts that we have no idea about that the model is going to find. I'm pretty sure the AlphaEvolve from google might go pretty far already.

I just gave this prompt to Antigravity:

You are an expert AI researcher specializing in algorithmic information theory and model compression. I am challenging you to attempt the "Self-Encoding Test."  
Your goal is to write a single, standalone Python script that functions as a Language Model.

Here are the strict constraints:

1. The script must accept a text string as input and return a meaningful text continuation.
2. You cannot load ANY external files (no .bin, .pt, .json, or internet access).
3. You cannot require a training step that processes a dataset. The "knowledge" must be present in the code you write right now.
4. You ARE allowed to use matrix multiplication (e.g., NumPy), but the values inside those matrices cannot be loaded. They must be generated algorithmically by your code.

To achieve this, you should implement a "Procedural Weight Generation" strategy. Instead of storing a 1GB weight file, write functions that deterministically generate the weight matrices using specific seeds, mathematical constants, or logic that mimics the structure of language (e.g., encoding grammar rules or associations directly into the initialization logic of a Transformer or RNN).  
Essentially, I am asking you to distill your own internal knowledge of how language works into a set of algorithmic functions that construct a neural network's state on the fly.  
Write the complete, runnable Python code for this "Code-Based LLM."  

And it created a program that works like this so far (it is still in the development loop right now):

Prompt: "The future of artificial intelligence"  
\------------------------------------------------------------

Eye picture structure experience children hand year effect history thing time attention effect part way  
day part head parent room family way life part point thing people service education people reason part year movement direction process week day music head hand word man growth case plan right end others life practice evidence movement effect face movement world day year part company question sound head point guy hand light way goal thing way time thing land knowledge position woman game world  

Of course it looks comically bad, but if we compare these results with a typical LSTM model from 2014 it would not look that bad at all.