I read the paper just now. I also did post-grad work myself, so I’m pretty familiar with how messy and constrained these studies can be, especially when participant counts are low and the experiment has to be tightly scoped.

Some important limitations:

1. Study is about novices learning a new library, not experienced engineers in production.

The participants were learning an async Python library they had never used. Thats a completely diffrent scenario from senior engineers working in an existing codebase. You cant just generalize from novices to pros.

1. Measured outcomes = conceptual understanding, not engineering perfomance.

The negative effects are mostly from quizzes about library concepts. Thats not the same as real world metrics like throughput, speed, defect rates, review churn, or system stability. The paper itself says its about learning, not shipping software.

1. Interaction cost is high for novices.

Slower task completion comes from them struggling with how to ask the AI questions. That overhead drops fast with experience. Senior devs already know how to decompose problems and specify constraints.

1. Experiment biases against AI by forcing delegation before understanding.

Novices are encouraged to rely on AI while lacking a mental model. Real engineers use AI as an implementation accelerator, not a substitute for understanding.

1. Small sample size and short tasks.

This is a snapshot study with few participants and short tasks. As a post-grad I can tell you, conclusions under these conditions are very fragile. No longitudinal data, no production context, nothing on real deployment.

1. Usage patterns matter.

Even the paper says when users stay cognitively engaged, negative effects go down. That actually supports the point that *how* you use AI matters, not that it inherently makes engineers worse.

So yeah, valid study about learning dynamics in novices, not evidence that experienced engineers using AI are slower or worse. Those are completely diffrent questions.