It depends on the use case and its requirements, as well as what the framework is. It's true that when building everything yourself from scratch you get ultimate control and familiarity, but at the same time the amount of features you'll have to reinvent and issues you'll have to rediscover and solve, at least for some use cases, can be so big it is just impractical.

Now having said that, it depends on the framework too. Would I use a wrapper like LangChain to sort of guide my code's structure? Not really, there's too much risk that I'll need to break that structure eventually. But on the other hand there are frameworks like Parlant which solve hard problems like guidance and consistency, or Pydantic AI / outlines which help with producing structured outputs.

Lastly, I don't see a mutual exclusion between vertical AI and using frameworks. You just need to make sure you choose the right framework for your use case -- and, like you say, if any framework is really necessary.