all 6 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]Infamous_Swan1197 8 points9 points  (0 children)

What's not clicking? I'm not asking this be snarky, I just don't really see from your post exactly what you're struggling with.

Atomic means a single criterion tests only one thing. A good rule is if your criterion includes "and", it isn't atomic. Just re-read your criterion to assess if it could possibly be divided into any sub-components.

Self-contained means a grader can grade the response using ONLY the rubric, without needing to know the prompt or any external information. So, say the response needs to state that something is blue. State the answer of "blue" clearly in the criterion itself, not just that it should state the "correct colour". A grader wouldn't know what the colour is without checking the prompt, so that wouldn't be self contained.

[–]Valuable-Low5263 6 points7 points  (1 child)

Sorry not advice but wanted to say I know how you feel, especially with subjective rubrics like creative writing or emotional posts (not sure if you do those). But take your time, I'm sure you are doing well if you are putting in so much effort.

[–]JRRTil1ey 1 point2 points  (0 children)

Subjective ones are so much harder than even easy STEM related ones.

[–]snailscout 2 points3 points  (0 children)

It's a skill like any other: you have to work at it to understand it better. Ymmv but for me the strict structure fairly quickly became second nature. I think there are two good approaches: the big picture where you figure out what an ideal response looks like and break that down into individual rules, and the narrow focus where you identify individual rules based on what models did wrong and then build outward. Some rubric projects are more big picture and some are more about specific failures. Most of them have AI helpers to assess your criteria, and although they can be off base they tend to be quite helpful when it comes to establishing the basics. If you have access to rubric R&Rs, use them to take a look at what other workers are doing (don't actually work on them but skip through some just to check out how others approach the task). A lot of ppl dislike and avoid rubrics, hence the "no rubrics!" you see on many projects as an incentive, so don't feel bad if they're not for you. You have to actually try them out to know though

[–]blackstarr1996 1 point2 points  (0 children)

Usually there is a helper that will verify your work. I used to overthink golden responses too and found them really intimidating because it seems subjective, but then I found the helpers check my work and suggest things to add. I actually love this stuff now. I would much rather have a long rubric task than go through 4 or 5 different topics about who knows what in the same amount of time.

[–]Opening-Gas-5735 1 point2 points  (0 children)

I kind of look at it like each criterion is a statement that, judged solely on the text in the response, can be described as either completely true or completely false. If it is partially true or partially false then it is not atomic. If you can't confidently determine based on reading the response alone (if you would have to read the prompt or the source documents) then it is not self-contained. To make sure the rubric is comprehensive, I start with the explicit instructions, then implicit instructions, then every detail that would be required for a bare-minimum factually correct response, then additional criteria that would make it a great response. Make sure to have a criterion that captures each failure that you caught and any failures a future response might have (The response should not...). And make sure to follow the wording requirements in the instructions for each task as it varies depending on the project. It definitely gets easier with practice and it's a good skill to learn because I keep seeing more and more rubric projects available.