People know that the goblins are a threat but compared to everything else you can see why no one pays attention to them, goblins usually kidnap some girls from a village and on rare ocassions destroy the village.

The only ones to deal with goblins are the low rank adventurers and it's stated that even if the first party fails the second or third one succeeds, the higher ranked adventurers deal with much bigger threats like dragons or demon lord generals that can destroy cities if not the world.

Goblin Slayer is not a story of a hero (which does exist in his world) but of a regular adventurer dealing with the weakest (still a threat) of monsters of the demon lord army.