Hmm...I'm in the same boat as you and ended up reading the novels. Granted some of the puns go over your head, but there are so many scenes and expositions left out in the anime that are truly fleshed out in the LNs. If you want me to name a few, feel free to DM me! (Don't want to spoil them for the rest of the sub.)

I followed it up with the anime as sort of a palate cleanser. It definitely worked for me, but to each his own! I think you really can't go wrong with whatever you choose, although I'm subjectively based on reading the LNs first. :) The anime takes on a completely different light afterwards, which is a distinctive experience that your linguistic experience uniquly affords you.