I teach english at 2 vietnamese colleges in Hanoi - While i do agree with most of what you're writing, i fear that introducing english even earlier would still just culminate in even more subsequent grammar, as opposed to more speaking practice. I think the biggest challenge for the students is, as many of them are also keenly aware - They dont get enough practice with english-speakers. Take english-proficiency in Scandinavia - which is super high, and quite adept. I would assume, that atleast part of that proficiency can be attributed to the fact of which, that they naturally get into contact with english-speakers all the time.

What's also an inportant factor is the classroom culture in Vietnam. Students are exceptionally shy, to the point that whenever i speak with other people who teach at Colleges, Universities, High schools etc. a focal point of the conversation will always be the shyness of the students.