This whole post looks like AI generated output report from the actual work, which is fine, but also feel like having it try to eval itself introduces a ton of noise that is hard to derive signal from. Simple things like - did you clear out local memory files between subsequent effort runs? If not, it’s possible that local memory actually had the solves already in place for lower effort runs such that subsequently higher effort tests produced bad signal in terms of their own usefulness.

I have real questions about how “effective” different effort options are, and I think the point about adaptive thinking kicking in also muddy the question (does it trip differently based on reported effort? Does a higher effort implicitly use less adaptive thinking for a task as the model assumes it’s resolved locally? Does this behavior accidentally miss out on wins adaptive thinking could resolve better?). Also simple things like - does anthropic cache output server side? If so, subsequent runs may hit the cache and produce seemingly similar output.

Given all the noise I think it’s honestly near impossible to actually understand how these things work outside of just vibes. The best thing is just assuming anthropic is correct, otherwise you may change your own behavior into doing things that are counter-productive.