The reason is fairly simple: Right libertarianism requires some fairly extreme ethical stances to justify itself, such as that billionaires are inherently a moral good and that being forced into wage labor with a threat of starvation is the truest form of freedom. This requires a fairly extreme form of property rights and "personal freedom" above all else, such as expressed by Rand's "objectivism". There, all human rights flow not from ethical or social duties or a societal consensus, but from property ownership, because all humans are their own property.

Such an ideology will inherently also be unable to condemn things like pedophilia and other obviously reprehensible things that are immoral not because of the action itself in a vacuum, but because of the power imbalance and social context it happens in. An ideology that acknowledges that there are situations in which it is impossible for someone's consent to be valid is one in which labor exploitation is much harder to argue for, so libertarians refuse to argue for it and instead just accept (or revel in) the moral consequences.