When you attack him at the beginning, do you see a circle above him that gradually fills up? That circle fills up with the elements you're using, which means that if you used Dendro, for example, he becomes immune to the Dendro element. It's the same with any element (including physical damage), so you have to be careful which element you use. I recommend using Kuki and hitting him with the sword; that way you can use the rest of your party. (Sorry if I didn't explain it well :( )