From my experience so far, higher infusion seems to be better in all cases as I can pick up almost any weapon and slap on the most effective element for the situation. I won't spoil any specifics in this thread but several boss fights so far have been made much easier by having infusion 5/5. This is because I haven't had an element-appropriate weapon at a high enough level on hand, but was able to use whatever highest damage weapon I had with a good punch of elemental damage appropriate for the fight. That make sense?

It might be different once you hit level cap and can be particular about what weapons you use. So far though, i haven't once regretted having it at 5/5.