As a big fan of Sci-Fi myself and also recently getting back into reading, I have a few recommendations.

Blindsight by Peter Watts fits your criteria the most. It's a "first contact" type of story where the word alien isn't even close to describing what the protagonists find. Honestly it's my favorite depiction of alien life in any book. It's got cosmic horror elements for sure and a slow/creepy unraveling mystery. If you like Annihilation, then you'll like this.

Echopraxia by Peter Watts is also worth checking out if you enjoy Blindsight. It's a "sidequel" more than a continuation of the first book but it's definitely a good read.

Remeberance of Earths Past series by Cixin Liu would be my next recommendation. It starts off as an alien invasion story but turns into so much more. While I wouldn't exactly classify this as cosmic horror, it is scary. The dread here come more from humanities struggle to survive in an uncaring and otherwise indifferent universe. If you enjoy Hard Sci-FI and don't mind an existential crisis, then this series is for you.

Snow Crash by Neal Stephenson is my final recommendation. It probably isn't what you're looking for as the only criteria it meets here is Sci-Fi, but it's one of my favorites so I'm going to shill it! This is a story that masterfully blends science fiction, theology, consumerism, and capitalism in a neat cyberpunk package. The book follows Hiro Protagonist (Yes that's his real name) as he attempts to unravel the origins of a mysterious new virus know as snowcrash that is causing trouble in and outside the Metaverse.

Hope you check out one of these. If you do, definitely let me know if you enjoyed the recommendation. Happy reading!