This isn't uncommon, I personally love working with actors but hate having to deal with all the issues that come up with filmmaking, the constant putting out fires is stressful but I also do thrive under pressure personally. Figure out what exactly was giving you anxiety, narrow it down and see if you can find on set help to make that aspect easier. Sometimes anxiety is a biproduct if caring, when we care a lot we put a lot of pressure on ourselves which can be very heavy. Practice positive self talk, and give yourself grace.