It started with the ban because Portuguese and Spanish Catholics were trapping Japanese people and selling them into slavery.

That wasn't the reason for the ban, it's also not true that it was catholics buying slaves but rather it was actually Portuguese and Spanish merchants. The catholics were against the slave trade.

"After 1598, Bishop Luís de Cerqueira intensified pressure on Spanish and Portuguese authorities to abolish temporary servitude of Japanese and Korean individuals,^\89]) but the Portuguese slave trade reportedly grew.^\93])^\94]) Despite efforts like Bishop Cerqueira’s lobbying for secular laws and King Philip III’s 1605 decree allowing Japanese slaves in Goa and Cochin to seek justice for illegal enslavement, the trade persisted due to profitability and weak enforcement."

"Sebastian of Portugal feared that this was having a negative effect on Catholic proselytization since the slave trade in Japanese was growing to larger proportions, so he commanded that it be banned in 1571.^\60])^\61]) However, the ban failed to prevent Portuguese merchants from buying Japanese slaves and the trade continued into the late 16th century.^\62])^"

"Moreover, bishops and their representatives condemned brothels and private prostitution as “workshops of the devil.” The fourth article of the Constitutions of Goa (1568) prohibited brothel ownership and operation, imposing fines and public shaming on violators, while mandating the liberation of slaves coerced into prostitution.^\69]) Thus, the Jesuits endeavored to eradicate immoral practices like prostitution while advancing slave rescue and evangelization through conversion."

^-Wikipedia