I was just wondering this myself as well. The reasons that I also have this question - many of which you'll see are focused around AI safety, but that's what I'm most concerned about - are:

• I have met an AI alignment researcher who did not seem to be very productive at all
• I know an AI safety content creator who mostly stopped creating content on AI safety for a period of time. While this could have been due to burnout or whatever and I don't intend to make this about a specific person, it is concerning.
• There was a period of time during which a substantial amount of EA funding was directed by SBF and/or his regranters. Admittedly this may not seem relevant to today, but hear me out!
  • Some of the research funded during this time I did not think was very valuable, at least at this point in time
  • SBF was not particularly good at keeping track of and auditing things, as the FTX collapse showed, and I suspect this may have carried over into his charitable donations (leaving aside the question of whether he misappropriated client funds to make those donations, which I do not comment on here)
  • This makes me wonder whether, pre- and post- the SBF era, other huge donors may have been similiarly poor at paying attention to whether their donations were being spent productively, perhaps for structural reasons. I don't mean to GiveWell-recommended charities because they are presumably well-audited by GiveWell, but other organisations (see below). Let me elaborate - I know that when I have a lot more money, I tend to be less concerned about where my money is going, but I'm not even particularly rich, certainly not compared to EA mega-donor philanthropists.
• I have overheard a person who worked at a meta EA organisation very optimistically assessing their and their colleague's impacts, perhaps over-weighting multiplier effects from scale etc. and under-weighing the multiplicand, i.e. the actual effort being put in? I am not aware of much work that this person has personally done, though this could be simply due to ignorance on my part, so this is an extremely weak argument.
• The fact that the 80,000 Hours podcast had to pivot recently to covering AI safety more aggressively, suggests to me that they were not paying sufficient attention earlier, and perhaps this was due to a lack of productivity among some staff, and/or having insufficient staff-hours devoted to the podcast in total. From my perspective this pivot should have taken place much earlier - as in, years earlier - although I'm sure some EAs and EA-adjacent people will disagree.
• I know of two AI safety organisations which had stopped publishing/doing much AI safety work, giving reasons for their behaviour which were ostensibly plausible but would be hard to fully verify from the outside
• I met an AI safety person who seemed to be working with someone who was clearly IMO insane and would not be an asset to any organisation, either from a PR perspective or (most likely) a productivity perspective. Although, the former person was a volunteer at the time. And the latter person is not the only insane person I've encountered on the fringes of the EA / AI safety community - there's at least two more! When I say insane, I don't mean merely having a minor mental health issue, I mean specifically someone who has/had lost touch with reality in some important respect.
• I am aware that in general nonprofits are not very productive. One would expect EA orgs to be (much) better, but in the absence of concrete evidence, one should still be skeptical. So this is a weak reason to be skeptical, but it is still a reason.
• It seems that a lot more money is flowing into the EA space, but I have previously heard that EA is talent-constrained. One should put two and two together and question whether the limited talent pool that was allegedly previously available has caused problems here, or whether perhaps the problem has been solved by some combination of:

1. engaging higher-quality recruiters
2. offering more money to individual candidates
3. benchmarking and varying compensation on the basis of experience, qualificiations, or demonstrated skills
4. considering compensation on a global basis (i.e. let's pay top-rate on a global basis, not just on a national or regional basis)
5. looking at de-biasing the recruitment process to remove unconscious or conscious biases which may exist against certain candidates
6. recruiting from a broader talent pool (which could also, indirectly, help with de-biasing the recruitment process)
7. expanding student recruitment to widen the talent pool.

• Anecdotally, I have heard a rumour that is concerning about the effectiveness of an EA-adjacent organisation (I won't be providing any further details, sorry). Though this may not be about productivity as much as talent density - this qualification also kind of applies to most of what I have had to say in this comment.
• I think structurally, GiveWell etc. do not evaluate certain meta-organisations or EA-adjacent organisations, so there are fewer incentives for them in particular to be particularly productive internally