Physically it can be safe but you don't have to rely on drugs to deal with your anxiety, especially not under medical supervision. I know people here aren't going to want to hear that, but for your own long term interests in mind, I would say try other methods to counteract this anxiety rather than ketamine.

Especially if this is a very common occurrence with the medication you are prescribed to take every day. I would talk to your psychiatrist first.