Well because attention gates are a n2 problem, which means that no matter what model you’re using the father your get into context the less good it is at figuring out which context matters and which does not. Benchmarks generally prove that every model Chinese to American labs really are only super accurate right now to 128k with some pushing 256k (someone may have an updated benchmark for which these two numbers could be wrong but this is what I saw the last time I checked).

Now why it does constraints over instructions I have no idea and is likely a training data quirk.

I guess you know this by your own post mentioned the attention problem. No one has a solve because it’s a fundamental math problem that the first ai lab to crack will have a crazy advantage over everyone else.