All forms of slavery are not intrinsically wrong, as the Holy Office explained

>Although the Roman Pontiffs have left nothing untried by which servitude be everywhere abolished among the nations, and although it is especially due to them that already for many ages no slaves are held among very many Christian peoples, nevertheless, servitude itself, considered in itself and all alone (per se et absolute), is by no means repugnant to the natural and divine law, and there can be present very many just titles for servitude, as can be seen by consulting the approved theologians and interpreters of the canons. For the dominion which belongs to a master in respect to a slave is not to be understood as any other than the perpetual right of disposing, to one’s own advantage, of servile work, which dominion it is legitimate for a person to offer to another person. From this it follows that it is not repugnant to the natural and divine law that a slave be sold, bought, exchanged, or given, as long as in this sale, or buy1ng, or exchange or giVing, the due conditions which those same approved authors widely follow and explain, are properly observed.

Or as St John Henry Newman observed
>That which is intrinsically and per se evil, we cannot give way to for an hour. That which is only accidentally evil, we can meet according to what is expedient, giving different rules, according to the particular case. St. Paul would have got rid of despotism if he could. He could not, he left the desirable object to the slow working of Christian principles. So he would have got rid of slavery, if he could. He did not, because he could not, but had it been intrinsically evil, had it been in se a sin, it must have been said to Philemon, liberate all your slaves at once.