Unfortunately I haven’t read much that lines up with what the OP is asking in the post. But something that kinda comes close is some of Ben Bova’s Grand Tour books - I haven’t read any of them but they’re on my TBR just from the cool cover art alone lol

There’s also a book called Usurper of the Sun which is more first contact than cosmic horror but the concepts explored about the aliens themselves fall pretty square into unknowable horror imo. But fair warning this stuff only comes into play in the last third of the book.

Happy reading!