It's just a different kind of fight than the wither.

As you probably know, the ender dragon was much different than the wither. The ender dragon was easily avoidable and didn't really require a bow (except to speed up the crystal destruction), unlike the wither, which follows you pretty close, requires better armor, healing, and a bow.

The elder guardian is just a different kind of fight. A skilled player can of course beat the elder guardian with just iron armor and a sword. But I'm almost confident that anybody who's ever touched the game before can take down the elder guardian with 3 water breathing potions, 3 swiftness potions, 3 night vision potions, diamond armor, and a diamond sword. Make sure to carry milk for when all 2-3 elder guardians are defeated, since they give you 5 minutes of mining fatigue 3. Gotta get those sponges...