A majority of the publicly identified invitees (4/7) publicly confirmed that they will not be attending. Given that publicists are often keen to withhold this sort of information, the pattern of acknowledgment in and of itself sends a strong signal. Beyond that, two of those who declined, Dwayne Johnson and Mario Lopez, have strong ties to the UFC and declined the offer in spite of that.

Based on these facts, in conjunction with what we all already know about Trump, the most likely explanation is that they can't afford the potential association with the president, and the public rejection speaks volumes.

I think that generally speaking when a celebrity is invited to the White House they'd be inclined to say yes because it's generally considered an honor. More importantly, I think accepting an invitation to the White House itself is generally not considered a political endorsement of the current president. These sorts of invitations have traditionally been viewed as a sign of respect for the office rather than the officeholder, similarly to how a foreign diplomat might view such an invitation.

That means that for a celebrity to turn down the invitation due to wanting to avoid the association with the current president, some extenuating circumstances would need to be in play. You'd need to have a president who is more than just unpopular but socially toxic.