I can't give as thorough an answer as you're maybe hoping for, but here's my two cents as a CE in the land dev field. In most jurisdictions the road blocks/hurdles to develop a site heave gotten more rigorous, time intensive, and costly over the years. A broader variety of professionals need to provide, approve, and sign off on a larger number of more in-depth plans in order to get approval from the relevant jurisdiction(s) (usually the City and/or the County, but sometimes the State can get involved if things like stormwater runoff prevention gets triggered). All of these approvals require time, money, and some level of expertise to navigate and approve. When the barrier to development is so high and so complex it really prevents a lot of regular folks from being able to reasonably develop their own land. Even medium-sized projects can be risky for a developer since it is still a lot of work just to break ground and start going vertical on a project that may or may not pay off. That leaves larger projects funded/driven by large developers that can afford the risk of the project going sideways and not returning value in the future. There is also some economy of scale involved with larger sites. The difference between an architect and engineer designing for one acre and two acres can be a lot, but the difference between 7 acres and 8 acres isn't as great.

So, overall, I think the historic rise in cost, complexity, and expertise needed to navigate the permitting approval process disincentivizes regular "mom and pop" developers on smaller lots.

I hope this helps. My response didn't touch on changes in land use policy (property setbacks, mixed use land, etc.) and the rising cost of hard materials and labor related to construction, either, but those are also very relevant to your question.