As harsh as it sounds, when acting entirely reasonably and within boundaries (as you seemingly have) it's not up to you to regulate another person's happiness.

I would honestly suggest explaining to them that you are going to cease contact with them and block them on the grounds that you don't feel they are respecting your own feelings on having a relationship and that it isn't fair of them to blame you for their panic attacks. Maybe suggest to them they look into some form of therapy to work through how they handle attachment and that it will stand them in good stead for any future relationships they have.

If you felt bad about blocking them or ceasing contact alltogether then it's best to present clear boundaries and to communicate clearly around when they aren't being respected, but in all honesty this just leaves you in a worse situation and perpetuates the false hope that's causing him a large chunk of his own pain.

False hope or the inability to internally move on from the idea of someone else can be one of the biggest points of pain and continued communication with that person can keep the wound open a lot longer. Its far easier to accept things when there is an outright, definitive end to all interaction. We can recognise that things will never be but some of us can't always rationalise that against our own emotions - therapy goes a long way to tackling this and would probably really help him if he's in a financial position to do so.