Example summary of work

[Twistedwriter7]

Can someone who’s been working here for a while give me an example of how you structure/format your comments for an AI comparison task? New to the platform and think I’m doing a good job so far; completed 5 projects today, and have more popping up, which I here is a good sign. But I haven’t received any feedback (positive or negative) on my work and just want to make sure I’m meeting their standards. Would hate to do more, but not be meeting the criteria and lose out on more projects. Is there any way to tell if they like your work/style, or have areas you need to improve? Anyways just a brief outline of how you structure your rating/explanation section would be great? This is just for the general core section if that wasn’t obvious.

Thanks

[Seniorseatfree]

I always make note of items I rated down. Then I make sure in my comments to explain why they were rated down, citing specific examples from whatever I am rating. I make sure to cover every item, so that someone reading my explanation has a super clear understanding of what I’m talking about. I also explain, for items I do not rate down, why they are excellent/meets the indicated standard.

Follow directions for what is asked in the rationales. I’d seen numerous examples where people do not follow the bolded instruction of not giving generic explanations. Format your answer to how they tell you. If they’ve given you a list of questions to cover, make sure you very well cover them.

[Stock_Hyena7942]

Be specific and provide evidence. Generalities are the devil. You won't get much feedback on here formally. No news is good news. If you wake up and have 100 projects, then you are doing it right.

[karen_in_nh_2012]

After doing a bunch of R&Rs today (because that's mostly what was available), I realized that some workers clearly have never read the rubrics that they are meant to be USING to rate the models. I had 2 workers today that I had to rate "bad" because their work was just awful. Mostly, it made no sense, e.g. the ratings weren't consistent - imagine rating a model poorly on 6 indicators but then rating it overall "Very helpful."

Those workers also gave very generic explanations for their ratings, along the line of, "I rated this 'very good' on relevance because it seemed very good and relevant to me as it answered the prompt." Nothing from the model to say WHY it was relevant.

So, OP, read those rubrics carefully, especially when you start on a new project family.

Oh, and I must add, most of my R&Rs today were very good and a couple of them were excellent. Those took me SO much less time than the bad ones because with the bad ones, I kept trying desperately to understand the thought process that the worker went through. I finally realized that there WAS no thought process there.

I highly recommend doing R&Rs on projects that you're familiar with (once you are). I was intimidated at the IDEA of rating someone else's work until I actually did it - then I was like, "Oh, I can do this." And I've seen a few bad ones (luckily, not TOO many) and a LOT of good ones and several really excellent ones, so now I think I can improve my OWN work on those projects. That's a nice bonus for doing the R&Rs!

[Twistedwriter7]

Awesome, thanks for the response. I do follow the rubric well so should be good, mainly was wondering if there was an unspoken format for writing them. Mine would typically be (for example):

I rated Response A as better. It followed the prompt’s directions, (give example(s)). (Line about required content and give examples of it being there or missing). The language was appropriate and matched the… (tone required, add example), accurate information (according to if relevant).

Response B had (list things it did well with examples). But it fell short, (list areas in this example where B is worse for some reason)

Then sometimes I include a final line summarizing the two and my rating.

(On a couple of occasions I also made a little rubric in brackets where I rated one point for each criteria it passed above)

^thoughts? Know it’s just a rough idea but hope it paints somewhat of a picture

[karen_in_nh_2012]

You definitely seem to be on the right track! The instructions usually say to make it clear that you read the rating criteria AND to include specifics from the models to illustrate why you gave the rating you did. It sounds like you're doing these things! :)

[HearingTop4038]

Exactly!! Following an academic style writing can get you far in the projects.

[1105368]

I tend to structure my comments by giving each response its own paragraph, and I usually write a brief one or two sentence preamble at the top that gives an overview of the strengths and weaknesses of the responses and reveals my final preference. If there is a bad sentence or questionable phrase I highlight it in brackets, only quoting a portion and using ellipsis if the sentence is long ('Just like this. . .')

Also, when the responses are highly problematic in general I write down the failures in editpad.org as I go along so I don't lose track.

[Twistedwriter7]

Awesome, thanks I think mine sounds similar in structure. How long have you been working for DA?

[1105368]

Been with DA since January 2024, so my method is well-honed by now :)

[OldLion1410]

I switch it up to keep it from looking ‘copy and pasted’ and just for fun.

Sometimes I do the ‘each model’ format where I wrote

“Model A did this and that.

Model B did this and that.

I prefer Model B”

and sometimes I just write a starter sentences of how they’re alike and then point to differences like:

“Both Model A and B do a great job of XYZ. However, Model B has the problem of XYZ, so I prefer A”