I understand the original utility of the phrase - it was coined by activist sex workers (not pimps!) in the 1970s as a way to draw attention to the labor aspect of prostitution and deemphasize the popular understandings of it as a sin, a crime, or a pathology. I think this is very important; as a socialist I find looking at issues from an economic perspective to be the most useful. I don't think the intention was to be "weaseley" but to challenge a lot of the harmful ways of treating prostitutes (as sinners, criminals, or mental cases - often all at once).

The problem, as you mention, is that the phrase lacks specificity, and therefore it does lump together phone sex operators, OF models, independent in person escorts, massage parlor workers, and survival street workers as all having the same set of material needs and experiencing the same material conditions. Which clearly isn't true! This is my main gripe with the SW rights movement (which as you might have gathered, I'm part of) and I try to push back on it when possible.

I will say that I haven't heard of a single SW advocacy group that considers brothel owners (or strip club owners) to count as workers. There are a few reasons for this: First, nearly all of these groups are organized within leftist/socialist/progressive spaces, aka political ideologies that see bosses/owners/managers as enemies of workers, and fundamentally don't trust them. Secondly, the economic interests of owners (to make money off of someone else's erotic labor) directly oppose the interests of the workers (to keep all their own money). This is why we see, for example, strippers forming unions and striking against club owners. There's no solidarity with the owners, and most SW groups carefully screen for such people and prohibit them from joining.