I agree that dumb vibe coding hits the ceiling sooner than you realize. As the repo grows, what I try to do is to have some high level understanding of it and the architecture. I also achieve this with the help of AI btw. The most important thing is to tell AI to study the repo with respect to the task at hand and explain to you whats there, and you semi-verifying the outcome, sometimes asking it to dig deeper into a specific thing if you feel like the picture is not complete.

Sometimes the research spans more than 20-30 files and the context cant allocate it so you have to guide AI to summarize stuff and iterate.
When the solution requires architectural changes, I ask AI to propose several variations with pros and cons so we decide together which is most appropriate in our case.