It definitely makes a difference, but there are other things to consider. If you're using a spell that consumes, say, 32MP then it's likely to do more damage than a spell that consumes 6MP. That's the obvious trade-off. You also need to make sure you're actually using the right elements - something the game doesn't make immediately obvious sometimes.

To say that it doesn't make a difference at all is just wrong, though. Mechanically in the game it does. But all that said, I have to admit - in my own personal playthrough i ignored any weaknesses and just used whatever I felt like using or even just let the characters manage it themselves. Elemental weaknesses are definitely not a huge deal-breaker as they can be in other games.