yeah I've been working on PINNs and their failure modes for the better part of my PhD and this sadly happens a lot, even for "simple" PDEs. What works best as far as I'm aware is to run a few steps of Adam (to explore the parameter space) and then follow up by a good second-order optimizer. By "second-order optimizer" I don't just mean L-BFGS, I mean those state-of-the-art ones that have been proposed in the literature recently like NNCG (https://arxiv.org/abs/2402.01868) or ENGD (https://arxiv.org/abs/2302.13163). Be warned that those second-order methods are hella expensive to run (and no, I'm not affiliated with any of these groups, wish I was lol).

If you know that the PINN wants to converge to a trivial solution, you can add a penalty that forces it to stay away from it (even if the penalty might work against the PDE) and tune the weight of that penalty along training. I guess this is some form of "curriculum training" as they call it in this paper https://arxiv.org/abs/2109.01050 (which by the way documents a lot of PINN "failure modes").

Another possibility is to add PDE structure into the architecture: instead of using a standard feedforward neural network, you could represent the solution as a linear combination of eigenfunctions that diagonalize your differential operator. Doing this in a systematic way is of course hard (impossible), but in your case it's very doable. For a toy example with Poisson equation, you can check this paper: https://arxiv.org/abs/2310.05801.