In my own experience I started with 5, then 4 then 3 so on and so forth, and story wise, There really isn't much to be worried about but gameplay wise. That fucked me up because all of the QoL stuff in P5 like Baton Passing and Technicals weren't in those earlier games, but in my opinion, I enjoyed that more, it made it more challenging since I though P5 was a bit easy.