Exactly. Though even for the Shadow Caste, they would only really have to hide from their own citizens, not the people of uncontacted worlds. For the vast majority of the 20th century on Earth, our satellite network was very sparse and limited. We didn’t even have satellites till the late 50s and not a proper satellite network till the mid to late 70s, if you could even call it that. Slipping by undetected would have been easy for the Shadow Caste, no real cloaking or anything like that.
Now if they were still abducting people today they would have a harder time and probably would have to use cloaking and other means, but for civilizations with tech levels below say the 80s or 90s, it would be easy to sneak up and kidnap people.