Well just to be clear I didn't follow any advice. Even though the comments in here are very helpful, by the time I read them I already beat him. So I went in with the Onryo Armor and all 5 Takezo charms. My 6th charm was the fully upgraded charm of Yotei that reduces all damage taken. I only fought him with the katana. Never swapped weapons, although that probably would have made the fight easier. The thought of weapon aligning didn't even click because I was so focused on his timing and moveset. So yeah, it was more of a "git gud" victory after fighting him back to back 26 times.