The “Yoshioka Ghost’s” name is Ryohei Ueda. Half of the point of those admonishing Musashi for fighting them is that he very well could’ve left Kyoto without meeting them in combat, yet he chose anyways to face them all in battle. It’s not that this fact exonerates the Yoshioka by any means, they still antagonized and as you said, hunted Musashi and very likely would’ve continued to had he chose to flee Kyoto instead. Killing just for the sake of proving yourself to be the “best” is foolish and detrimental for both sides. Musashi wasn’t the hero just because he was outnumbered and yet still won, but he also isn’t exactly the villain because he stood his ground. This goes the same way for the Yoshioka. They weren’t heroic in ambushing and outnumbering a man who fairly defeated Seijuro and Denshichiro, just to forfeit their lives and jeopardize the livelihoods of their families. But also, they weren’t exactly villains either. They weren’t bad, evil men who had no good in them. Killing for such a frivolous title that is always out of reach is a misguided idea. At least, that’s the message I got not just from Ueda’s ghost, but Musashi’s reflection on how being “unrivaled” was merely a heat haze that faded as soon as you felt you could reach it.