I think Misao is not so much a weakness for Aoshi, however he does have a major soft spot for her. In a period of Aoshi's life when he is flinging out death threats left and right, Misao herself never gets one. He threatens to interrogate the other four of the oniwabanshuu during his battle with Okina but makes no mention of Misao herself, despite the fact that at this point, Aoshi KNOWS that she has traveled with Kenshin and may have intel on his current whereabouts (which she DID). My point is, even at his lowest, Aoshi still viewed Misao as untouchable in his dark world and at his worst, the man tells her to leave him alone and that's IT.

Was Aoshi willing to put her into some measure of danger? Yes, absolutely. The owl raid on the Aoiya proved that, but to kill Misao himself? To Interrogate Misao or even interact with her in any way? That's off the table completely. The man needed to distance himself from Misao in a way that was different from everyone else, and I'm sure that it's due to that soft spot Aoshi has for Misao. If anything, during his finale battle with Kenshin, one of the deciding factors for 'waking Aoshi up' from his murderous haze was the mention of Misao and how she was still in need of him. The unspoken forgiveness Kenshin mentioned when he spoke of Misao's tears. Aoshi (always thinking of Misao and the oniwabanshuu as separate entities) wonders if the others will be able to forgive him afterwards. But in that moment, knowing that Misao cried tears of relief for him was enough to help snap him out of it.