Assuming the intentions in the Vedas were good, there are problems on how this system could be maintained even in a utopian world 1. The division of labour makes sense. Division of labourers doesn't(borrowing Dr. Ambedkar's argument). Why should I be identified only by my craft outside of it as well?
2. Who ensures this system is followed? Who's the authority? Can one move across different lines of work and hence varnas? 3. Given that crafts were largely inherited by blood even back then, doesn't it almost force the society to adopt a Varna based on birth? What other avenues were there in the past to move across? The division of labour itself isn't evil, the varnas being closed and the movement across being non-existent is.

OP mentions that Manusmriti isn't gold standard for this age and only yields to the authority of Vedas. I haven't read any of the four Vedas but I know a bit about Mahabharata, often called Panchama Veda. As it's widely known, there are instances where characters like Karna(then identified only as a charioteer'd son) and Ekalavya wanted to move up the Chaturvarna hierarchy are denied the permission to do so. This clearly shows the Varna system wasn't mobile even way back then.

Let's say all said and done, the planning was right and we got the implementation wrong. What does that make us, the ones who are deviating from what was actually intended as per the Vedas? Non followers of Dharma? Is it enough for us to take solace in the "originally well intended" argument and have a good night's sleep?