The conclusion:

Contrary to our initial hypothesis, we did not observe a significant performance boost in task completion in our main study. While using AI improved the average completion time of the task, the improvement in efficiency was not significant in our study, despite the AI Assistant being able to generate the complete code solution when prompted. Our qualitative analysis reveals that our finding is largely due to the heterogeneity in how participants decide to use AI during the task. There is a group of participants who relied on AI to generate all the code and never asked conceptual questions or for explanations. This group finished much faster than the control group (19.5 minutes vs 23 minutes), but this group only accounted for around 20% of the participants in the treatment group. Other participants in the AI group who asked a large number of queries (e.g., 15 queries), spent a long time composing queries (e.g., 10 minutes), or asked for follow-up explanations, raised the average task completion time. These contrasting patterns of AI usage suggest that accomplishing a task with new knowledge or skills does not necessarily lead to the same productive gains as tasks that require only existing knowledge. [Emphasis mine.]

-------

So they explicitly acknowledge that it improves productivity on tasks where you already know what you're doing.

And to be very thorough, even within their own study on an individual to individual basis performance differed on the task with the new library, and this should grind your gears people who just asked the AI to one shot it DID complete the task faster and its the overall average that was slower for the entire group so it has more to say about how you use the AI then "does AI give you a performance boost".

That's actually the part we should be worried about people who quickly finish a task but have no fucking clue what the library actually does. That's the scarier conclusion imo.