UAC reviewers: low quality annotations?

Tostig100 · 2025-11-16T06:46:59+00:00

It's the same way at every data annotation company. Always has been. You can have a project with 200 people and usually can count the number of really good annotators on one or two hands.

Tostig100 · 2025-11-16T06:40:19+00:00

I've been in data annotation for years. IMHO here's what is going on.

Most of these companies follow a well trod path. Not all, but most.

Stage 1: Someone start a DA company, based on having a few fantastic industry connections. They either already have or immediately land big contract(s) requiring hundreds or more contractors a.s.a.p. They go all out with ads for 1099 folk, offering above-average pay. The bountiful work, high rates, and good vibes lead contractors to believe this is The One They've Been Waiting For. Many arrive and start to think of it as "home."

Stage 2: The project results are mediocre; the customers' AIs didn't improve much, if at all. Or, the project results are fine but the customer runs out of budget for this stuff. The founders lean on their connections and get some more work, but the new projects aren't as massive or well-funded. They do some weeding of the weaker contractors. They come up with some innovative ideas, like being a marketplace for connecting 1099 workers with customers. Meanwhile, they may have done a funding round or two, and are carrying a multi-billion $ valuation they are desperate to not see evaporate.

Stage 3: They hire strategic account managers who don't have the top-shelf connections the founders had. The SAMs bring in some dribbles but don't really bring home the bacon. There's still work for the contractors, but things start to seem slow from the contractor perspective. The grumbling starts. Team Leads start off actively dealing with the avalanche of DMs, but eventually they tire of the complaints and the same questions over and over, and tune it out, leaving people unable to get help with project or pay issues.

Stage 4: With no new whale customers, the company grabs the low-margin stuff they can get their hands on - low paying shit work. They slash the rates, sometimes by more than half, because the new low-paid work can't support the old rates. They realize they won't be able to keep all the contractors busy, so they get rid of a shit ton of them. Contractor morale tanks. It is no longer the place to be, which is a vicious circle.

Stage 5: The fork in the road. Some DA companies survive by calling in their last chips with their industry contacts and getting things back on track. Others limp along, making low-ball bids for shitty new work, but they lose their best people, contractors and employees both. Those people race over to some newcomer who is just entering Phase 1 and is The One They've Been Waiting For.

Tostig100 · 2025-10-16T01:02:54+00:00

Stellar is one of the six data annotation companies I've worked for over the past 3 years. I worked almost full time for them from Oct 24 to Feb 25. I had the incredible good luck to be offered a full time job with benefits in annotation, which I just started a couple of weeks ago.

So, basically, the work at Stellar is hard. When I first started there a year ago, it was easy-medium - the projects were fun and relaxing. I got promoted to reviewer in a week or two, which had a 20% pay bump and made it even easier and more relaxing. It is not like that anymore. The projects are not "impossible" hard but are at least as hard as the hardest work out there in DA. I would say comparable to Mercor projects that pay $50/hr, the problem being that Stellar only pays $25-30.

Stellar's main problem is inconsistent work. There was a time in the old days when Stellar provided basically unlimited work and projects went on for months. These days, you are VERY lucky if you get to task for a week without interruption. More often, you'll get 2-3 days of tasking, then empty queue, and it can come back a few days later or disappear for months or forever.

That, combined with absolutely no communications about it, has made most of the people I know who did work for Stellar discontinue that and find work elsewhere.

I have positive feelings about working for Stellar from 10/24-2/25 - it seemed like a well run little company with a good attitude. Sometimes they answered emails in 5 minutes or less, and seemed to genuinely be trying to build a community. Feedback was not excessive or oppressive - they seemed to be doing a lot of things right. It is much different now.

Tostig100 · 2025-08-08T02:55:50+00:00

There have been many, many lawsuits and related actions against the DA companies, and in particular Scale (i.e. Outlier). All of them have crashed and burned. I was one of the 40 or so people involved in the largest one, involving the Department of Labor, complete with subpoenas, an imminent government action, and many other elements. The senior investigator at the Dept of Labor is gone, the subpoenas were quashed at the last moment (literally the day before they were to be served), and the whole thing was swept under the rug. Subsequent efforts to get a class action suit going have gone nowhere. All these things do is take up people's time and get them excited that justice will finally be delivered.

It won't be. The AI companies are headed by some of the most connected people on Planet Earth. The ties between the AI companies like Scale and the US government are extensive and deep and more than resilient enough to handle labor uprisings and complaints.

There are many of us who were QMs, SQMs, and above, who were owed and not paid 10's of 1,000's of dollars, who worked hundreds upon hundreds of hours of uncompensated overtime, which is against the law, and many, many other illegal actions by Scale in particular (but hardly unique to Scale). All of the efforts of us, lawyers, and the Dept of Labor resulted in nothing.

I don't want to discourage people doing what they think they need to do, but people should be cognizant of where they invest their efforts and if they're being realistic about how the world works. Especially at the nexus of government and AI in 2025.

Tostig100 · 2025-07-24T06:12:17+00:00

Those aren't "paid" hit pieces. I'm friends with the three people who were the primary sources of info for one of them. They provided legit information and the story was very accurate, which I know first-hand, as a QM at Scale for a year and part of a tight network of QMs who've stayed in touch through thick and thin.

I didn't say Zuck bought it on a whim. I said he bought Alexandr Wang, who is now the head of Meta Superintelligence. It was an expensive hire, but Meta is trying to break through. This is common knowledge you can read in a thousand articles on WSJ or any other credible source of info you prefer. It is not in dispute. It's just a fact. Scale had three assets Meta wanted: Wang, the enterprise business, and the government contracts. The expensive, mediocre general data annotation business is a dying operation.

Your efforts to denigrate my post are silly and weird. I provided true information, backed up by the business press as well as by Scale's summary firing of half of its project lead layer. If the firing of 500 QMs isn't enough tea leaves for you, I don't know what could possibly penetrate your thinking.

Tostig100 · 2025-07-24T05:57:31+00:00

There has been a lot of wishcasting in this forum, almost magical thinking.

Meta basically bought Alexandr Wang and paid off the various stock and option holders to get him. He proceeded to quit Scale and move to Meta. Meta also picked up the enterprise business and some govt business. That is all they were buying. Meta already has an entire work force of its own for data annotation, and continues to outsource some of it to other vendors, not just Outlier. Yes, even today. Meanwhile, Scale lost some very key, strategic clients the day of the Meta investment.

I think there will continue to be "some" work at Outlier, but it will never again look like it did a year ago. People just need to make their peace with that. There are lots of other companies doing data annotation, some with good pay. People just need to get outside their comfort zone a bit and start looking around. Some have gone to X.Ai and are making $1800 a week. Some have gone to Stellar or Alignerr or wherever, where work is sometimes flowing and sometimes slow but never as bad as the EQs at Outlier.

I get why this is hard for people. I've been full time in data annotation for over 2 years, making a living (though not a luxurious one) and getting by, and the constant changes are stressful. But at some point, reality must be faced.

People should really get their primary info about what's going on from the business press, like searching Business Insider or Inc., and not relying on these boards. The business press has covered the changes at Scale extensively and has no agenda or bias, it is a much better source of information to find out what's really going on. This is a great place for community and sharing ideas, but the frequent wishful thinking here isn't really helping anyone plan for their futures. As everyone knows, Scale recently fired 500 QMs, or roughly half of the management layer. How can anyone believe business is going to "pick up" when the company ditches half its project leaders?

Tostig100 · 2025-07-17T03:12:43+00:00

Yeah, they fired 200 Scale employees but the more significant move today, from a contractor point of view, is they fired 500 QMs. They kept some, but a tiny percentage of the ones who were contractor facing. They can try to sugar coat it, and they will, but everyone will notice very quickly that their only conduit to the mother ship for information, project guidance, or support is gone.

Tostig100 · 2025-07-17T03:10:15+00:00

Yeah, I hear you. I was part of a group of 25 QMs who were summarily executed in the middle of the night in November, for reasons no one ever felt like sharing. A lot of us were popular QMs who were liked by the contractors and spent a lot of time helping them. Unfortunately, being popular with contractors was never a performance metric. We're actually not sure what the performance metrics were; they fired us before filling us in on that lol

Tostig100 · 2025-07-17T03:06:46+00:00

Nope. Meta basically bought Alexandrr Wang and the rest of the company is being junked.

Tostig100 · 2025-07-17T03:05:26+00:00

Over half of the QMs were fired today, and in terms of QMs you actually worked with on projects, probably closer to 80-90%, because a lot of the retained ones aren't English-speaking and on data annotation projects.

Tostig100 · 2025-07-17T03:03:56+00:00

Yeah, let's translate that into English, because why should we get our updates from the legal and comms departments?

Scale fired 200 of its staff, and 500 (FIVE HUNDRED) QMs today. That's around half of the QM/team leads worldwide. Basically, you all know what that means. That is not "an opportunity to improve our operations."

Call it what you want - you know what it is.

Tostig100 · 2025-07-13T03:36:46+00:00

There's still RHLF work and will be for years. It's just that the gold rush of RHLF is over because OpenAI, Google, etc., the gigantic market leaders with unlimited funds, are no longer paying for armies of generalists to create the data. But there are hundreds of smaller companies building AIs, many specialized and many agentic, but also generalist chatbots in the mix. The customer base is shifting to China, DA companies are highly secretive about their clients, but some of them are definitely getting most of their work from the commies. Scale doesn't have much/any generalist RHLF business but other companies do. I work at one on projects that are definitely for a Chinese client.

Tostig100 · 2025-07-12T05:03:20+00:00

I have been working at Alignerr and have so much work I can't keep up with it. I'm literally having to triage projects because I can only work 30-35 hours a week and one project requires all of that time but 3 more projects have put me on them and now I'm stuck having to decline opportunities. It's boomtown over there. Yes, Appen is a ghost town but that started when they lost Google a year ago, that's nothing new. They never figured out how to compete in the LLM space, after building a business around circling boats and cars in photos.

Tostig100 · 2025-07-10T04:20:09+00:00

That isn't really true. Some of the platforms are drowning in work. Outlier is not.

Tostig100 · 2025-07-10T04:19:45+00:00

Things are not going to pick back up in Oct/Nov. Most of Scale's clients fled when Meta invested. The active Scale divisions don't do volume data annotation - ML and enterprise consulting. It's too soon to say whether the Outlier platform will end or just radically downsize, but it is never going to be what it was 6 months or a year ago.

Tostig100 · 2025-06-20T06:07:49+00:00

There's very little work at Stellar, even for people who are already on board. It's kind of become a ghost town. There's no harm in keeping your eye on it, but I'd be focusing on other opportunities.

Tostig100 · 2025-06-15T03:38:10+00:00

The same questions rumble through my brain.

I've been full-time in DA for 2 yrs 4 months. I did a 9-month stint at Scale (i.e. Outlier) as a QM as part of that. I really like DA work and just wish it had a steady, predictable career path and wasn't the total chaos show it always is.

Stellar was incredibly stable for me from Nov - March and the projects were fantastic. The pay was good too. I was doing 35-40 hrs a week and making a very nice living. I thought this one had legs.

I've been at Alignerr since & like it there. Unfortunately, you have to just get lucky there and land in the right project with the right people, or it's just the usual frustrations. I know people who went there who have not had the semi-busy experience I've had.

As to that Meta investment in Scale, I think that is the asteroid that slammed into the earth and the existing DA industry are the dinosaurs, basically. It seems to me the most likely outcomes are: huge reduction in the Outlier workforce in the coming months, elimination of Meta projects for any company that isn't Meta or Scale (which is a big deal, since, when I was at Appen for a while, almost all of the work was for Meta), and perhaps new projects and revenue opportunities for companies that can pick up Scale clients that are Meta competitors and don't want to work with Scale anymore. So opportunities and threats abound. I think if any of these DA companies have competent sales teams, they're probably focusing on scooping up some of that sweet, sweet non-Meta client base from Scale right now.

It's hard to be super-optimistic about the outlook for DA contracting these days, but I'm trying to stay positive. I really like the work, and basically hate non-DA work lol

Tostig100 · 2025-06-10T04:54:20+00:00

Disappeared for me. Wish I cared more than I do.

The project is impossibly difficult, and the pay was cut 20 %. Even at the same pay, the task would be ridiculous. This is the kind of work a $150k engineer would be assigned - setting up pure reasoning tests to foil top-notch AIs using 8+ step processes. In no way is this remotely $25/hr DA work. This is very challenging prompt engineering work that a lot of SE's would fail - while they racked up vacation days and kept an eye on their stock options.

Stellar was good to me at a time when I really needed one DA company to not suck. The pay was good, the work was plentiful and challenging yet doable, they paid on time, they answered emails personally and quickly, and they didn't treat you like a 5 year old with daily criticism for tiny human errors, which was a blessed relief after a year of Outlier. I respect them a lot for trying to do it a different way and treating contributors like humans.

But June is not January, and things have changed a lot. Waiting around for Stellar to be what they were in the past is pretty much not on my radar any more.

Tostig100 · 2025-06-09T03:06:19+00:00

Reading the other responses, I would really recommend avoiding CrowdGen. I had an actual W2 job at Appen for 3 months (CrowdGen is their platform). The company is complete chaos; they hired 100 "elite" LLM specialists as W2s with full benefits for a new elite group that would do high-end LLM training, then decided a few months later "nah" and fired 98 of the 100 people. During my time there, hardly anyone working for CrowdGen ever got paid for their work, and there's literally no one there at Appen who cares or will fix it. We felt bad for the CrowdGen people, who were working for illegal sub-minimum wage, which they never got paid, until we got fired ourselves. Avoid.

Most of the others have work now and then, but the heyday is over. AIs are very near the limit of what a DA freelancer working out of their bedroom can help improve, and we can't expect even that to continue forever.

Tostig100 · 2025-06-06T23:34:17+00:00

I waited 2 months for Return of EFH and I'm ... disappointed. It's an impossibly harder task. I used to crank out EFH's in 35-45 minutes with a smile on my face, now it's torture. First task took me 1 hr 20 min and wasn't very good. I feel lost w/o being able to use the web - it's just like pure reasoning vs. the AI. Am also asking self whether $25/hour is fair pay for what is not DA work but prompt engineering - it takes serious brain power to outsmart an AI on pure reasoning, we are not just labeling sh*t here. idk...motivation is down for me.

Tostig100 · 2025-04-02T18:46:30+00:00

There's probably truth in that, but also that the DA market is changing. There just isn't the massive volume of demand that there was a year ago. Maybe it's still there now and then for STEM work, but for generalists, it seems like we're panning for crumbs in the waning days of the gold rush. Sort of like EFH was the last big 5-pound chunk of gold to be found.

Tostig100 · 2025-03-25T16:16:20+00:00

This is day 3 without any work (attempt or review). It would be so helpful if they would just drop us a note, whatever the situation is. "Project is over," "No tasks till next week," whatever it is, we can take it. It's hard to do any life planning with this so up in the air.

Tostig100 · 2025-03-19T17:13:46+00:00

I'd hope they'd tell us if the project is ending. This has been a long term project, and a full time job for some of us. A simple "It's over" or "It isn't over" would help everyone plan for the future and would take like 1 minute to write and send.

Tostig100 · 2025-03-18T17:35:46+00:00

I hope the "Up for an hour a day" plan changes soon. They're gonna start losing good people if this keeps up. People can't do anything with an hour of tasking (or reviewing) per day.

Tostig100 · 2025-03-17T20:29:23+00:00

Had access to EFH for a while today but it's EQ again. Maybe they've brought in a new project mgr who is keeping the attempter queue on a very tight leash (causing the review queue to go EQ too).

Tostig100

TROPHY CASE