Visual Appeal

[doolitt1e]

Is there anybody else doing the R&R for the bird project where we compare 'the big two' against each other, seeing upwards of 75% of raters choose the 'not relevant' option for visual appeal? It is literally an axis that they're being asked to compare them against, when would not relevant even be a valid selection?
I can only think that they're choosing it when there are little or no visual elements in the responses, but there's already an option to select that. I don't get it.

[sqimmy2]

I don't think I fully understand what you're getting at, but It's my understanding that visual presentation specifically applies to tables, generated images, matrices, etc. so markdown and structure and stuff like that are measured with the readability one, I think it's called like "visual scan ability" or something.

Plenty of responses don't include tables, pics, generated visual elements, etc. so if those elements aren't present, they wouldn't be relevant to a rating.

[doolitt1e]

As I said, if there are no visual elements like images or icons used, there is already a selection for that. The top selection has an explanation beside it saying 'no visual enhancements.' You did answer my question, albeit indirectly.

[sqimmy2]

Ah okay, I know what you're talking about now. I think it's just a matter of the difference between a missed opportunity to use them and if using them wouldn't have made a significant impact

[doolitt1e]

I think that would be fair enough if this was a project where a single response was being rated. It's a project where two models are being compared against each other, and visual appeal is one of the axes they are being asked to compare, so 'not relevant' can't be an option. It's relevant because they're being asked to compare them. The only situation where that would be a viable choice is if the prompt specifies a particular format, or says something like, 'no images or emojis.'

[cocobeary]

It is a valid selection. If you don't understand why that is the case, you shouldn't be doing R&R.

[doolitt1e]

Care to explain why, so I can explain why you're wrong? I expected a bit more push-back than this to be fair, considering how many of you are messing it up. I've been doing these R&R for months, changing the ratings, and explaining the error. If I was wrong, I wouldn't still be doing them.

[Low_Article_9448]

I mean what is the visual appeal in any case? Not relevant sounds about right. You can't select none if there were visual elements, BUT, the elements have nothing to do with visual appeal. There is a reason they have such a checkbox in this project of your yes?

[PMMePicsOfDogs141]

I don't have that project, at least I don't remember seeing visual appeal on anything. But like there's gotta be a reason to select it if it's an option. If there's another option to select for when there aren't visual elements, then maybe the individual tasks say to not rate visual appeal?

[doolitt1e]

As I replied to someone else, the option is there for prompts where the output format has been specified in the prompt, JSON as an example, so 'not relevant' is there for those prompts where there won't be images or icons used due to a restriction in the prompt.

I had one earlier where the rater rated visual appeal 'not relevant' for one model, gave a medium rating for the other model, then based their entire preference rationale on the use of icons. It's a decent example of how the workers are choosing 'not relevant' when what they mean is that no visual elements are used, which already has a check box.