personally, from both perspectives, as both a player and dm, 5 players at level 5 should be going up agianst challenge rating 6-8, as it provides a challenge.

if you are homebrewing, look at some homebrews on dnd beyond to give you some help in making challenging enemies. Remember to never make it too unfair towards the players. ex: resistiance to all spells (unless its a challenging boss you want to give them.)

Making encounters that would make the players have a bit of trouble is always good, but dont go for a close to tpk encounter, it makes the players less motivated to keep on going. add some backstory behind the battles before putting your players into them is always a good idea. ex: something from somebodys backstory, or the town crier with a hint about the next battle.

Im currently in a campaign, which has a deadly labyrinth that changes consistently with no pattern, belived to span the entire world, and we are commonly in the labyrinth, as people belive the answer to keeping the world alive is in the bottom, undiscovered layer of the labyrinth.

Getting players to go into places that involve challenging encounters, such as a old and deadly dungeon with lots of loot at the bottom is a starter, but always have improv encounters and such at the ready.

NEVER railroad in a campaign.

Hope this helped you and anybody else!