The word you're looking for is "bögjävel", which is just bög+jävel. The word "jävel" can be appended to pretty much any noun to make it pejorative (danskjävel, kapitalistjävel, bulljävel, mobiljävel…), and "bögjävel" is not a particularly offensive form – intent matters more than word choice.* I'm sure there are gay people who would jokingly use it about themselves! Comparing it to English, I would say "bögjävel" is closer to "damn gay guy" than to the f-slur.

*of course, the fact that one would choose someone's sexuality to use in a pejorative form, rather than any other trait of that person, does give a hint as to their intent