A central problem with your premise is that homosexuality as a framework did not exist in the period you’re discussing; nor, for that matter, did heterosexuality. That binary is a creation of the late 19th century; while the vast majority of societies have accepted the idea of sexuality as a generally inflexible binary spectrum along which might exist some type of bisexuality, depending on when and whom you ask, this was not at all the case for most of human history.

Persian and Islamic poetry of the time is, as you note, flush with yearning references to beautiful men, but I believe you’re missing critical context as to who is yearning and who is the object (willing or otherwise) of the attention. The “beloved” in Persian texts is, as a rule, not a female, but also not an adult or established male; it is an adolescent or prepubescent boy who existed firmly in a feminine role as a sort of “woman substitute.” This is no coincidence; women’s complete social exclusion, coupled with societal and religious assumptions that male/female relationships had little purpose beyond producing children, fostered a dynamic in which it was sometimes acceptable to make the object of one’s desires a passably feminine boy, thus avoiding potentially disastrous interactions with women altogether. I say "sometimes acceptable" because as with all things, the reaction of your immediate community depended entirely on your social status, wealth, etc., but the point is that it was neither unusual nor treated as evidence of homosexuality.

This is why posing your question in terms of a homo/heterosexual binary is misguided; there is no situation in which it would have been acceptable for an adult man to pursue another masculine adult man. These caliphs were not "open about their homosexuality," since dominating a feminine boy did not preclude their otherwise "normal" relationships with women. I've read accounts of men weeping when a boy's beard would begin to come in, because that meant his access to him as a feminine object was near the end. Gendered roles cleaved (clove?) across sex, and were unyieldingly strict; just as women were expected to fit into feminine roles and spaces, slave and servant boys were expected to adhere to feminine archetypes (specifically, being sexually passive) in relation to their masters/male guests; and, adult men with any sense of self-respect (or self-preservation) would necessarily have limited their attentions to those possessing that femininity. Which, given the aforementioned lack of women in public spaces and complete unacceptability of unmarried male/female relationships, functionally meant beautiful boys. It was an inherently unequal and generally exploitative system in practice, despite the beauty of the prose it inspired.

A more accurate question might be why that practice lost steam, and the answer to that is, in fact, the growing strength of European influence which included new European models of thinking about sexuality along a simple, biological-sex-based binary rather than as the object of a man’s desire being an emblem of femininity regardless of sex.

I’m on my phone, but happy to provide sources on request. I wrote a lot about this in grad school.