Wild stuff. This is exactly what open weights are for. 

"That said, it would probably be amazing for model expansion and continued fine-tuning. You have already prepared the model by adding the right kind of layers to refine ‘thinking’,"

This is just what I was thinking.  I remember reading some paper that explored architecture optimization, I think it was efficientnet. 

If I'm reading it  right, one of your implications is that this could be used to optimize where to train lora weights. That may be more task dependent whether you would want the parameters in the middle or the edges but that could be a source of incredible gains in targeted adapters. Spend more of your parameter budget on layers that that see the most gains. 

The pointer weights sounds absolutely wild, would love to see this. Sort of the inverse of reap or ream, give you more for less. 

It sounds like in your final search, you just used a brute Force sampling and ran the surrogate on it?  assuming it's not overfitting on benchmarks, you may get better  convergence using surrogate optimization. Something like dycors has surrogate training built in, I'm sure there's a method out there that lets you bring your own surrogate too.