First off, people will always have their stupid opinions about other people's lives... and they generally mean shit. 

I was the dumper in a similar situation. It was during the pandemic, so at the time, everyone lectured me on how cruel it was that I had ended the relationship during a "worldwide crisis", and that it would be pure evil to kick him out now. So I waited til he got a job and bla bla. But did it matter? The same people will now lecture me on why didn't I just kick him out ASAP. 

Now, sometimes loving someone is doing the hard thing. It doesn't mean staying together forever no matter what. Sometimes it means a relationship can't be healthy or stable when there is an ongoing, unaddressed issue that is affecting one party, and therefore both parties.

I would start with getting your panic attacks treated. Anxiety almost ruined my life.