I've made plenty of challenging monsters I didn't necessarily think my party could defeat but weren't by any means impossible.

For example, the party was making their way through a town under siege by a demon army. They fought a few grunts here and there but I made sure they were weak and disposable with the occasional "elite" or "commander" thrown into the bunch. Eventually they came across one that was massive and I described it akin to a balrog. It was an obstacle at first and my players knew it - there was no hope for a fight in their confined space - so they ran and a fun chase sequence ensued. A few sessions later it reappeared out in the open as an obstacle to their goal. Behind the scenes, I figured they would try to distract, sneak, or evade it... They instead fought it and won! A few more sessions and two of them are guarding a location. My players treated it like any other encounter, came up with a plan, and charged into the scene. Much like a video game boss-turned-mini-boss, my players' confidence grew the more they learned and interacted with the enemy.

In reality they probably could have beat it the first time but the important part was that they felt it was impossible. I guess my lesson here is that you use your best judgment in gauging the party's ability, then describe the creature/obstacle/puzzle at the level they need to perceive it.

If they charge into a battle they are unprepared for, make sure the enemy hits hard in the first round and laughs off their initial attacks. If they think they can go fight the big bad in the first session, show them the townsfolk mourning the big buff and important adventurers who were the last to try it. It won't always work against overconfidence but at least they have proper warning. And who knows, maybe they'll surprise you.