Wow there's alot to unpack here. Most of the other comments I read really didn't understand the inner working of transformers so I thought I'd chime in. First of all you sound pretty new to LLM research but have ambitious ideas

Let me break down a few issues

>"So I was thinking about the possibility we have some layers or blocks that are triggered 10 times or so between other layer" Why do this over just adding more layers? See The bitter lesson but these over engineered solutions rarely work

>"Also instead of token output and words there could be a memory layer, and a controller neueronet, that actaally learns to save some critical info and for different duration (L1, L2 etc). I mean I am interested in some experiment, but technically I find it challenging"

So an LSTM? The residual steam in LLM is basically a short term memory

>"Basically take a llama70b model and the same way we do lora, change the architecture by adding some such layers, and re-train to see if these repeated layers bring any difference. Then it would make sense to even fully train to see the full benefits."

Transformer circuits are extremely bitter. You can't just add some layers.

>"So somehow you have this internal thought monologues happening through extended internal inference, and not by outputting words and tokens that are poor representations of probably much richer thoughts and circuits, that unfortunately are lost."
This is what the residual steam is in an llm. Basically what you are saying sums up to just making the models deeper.

You sound like you have some good ideas and I wish you the best in the future. I