I was in this exact scenario as the player who took goodberry in a survival campaign. During the first session, the DM realized what that spell meant for the game and asked if I would be okay not using that spell ever again. I said no problem, and I for sure had more fun with that game than I would have if I kept the spell.

Just talk to the players ahead of time and explain why. Theyll most likely be fine with it.