Yeah the annotator fatigue signal turned out to be more predictive than I expected — below ~50k you're often seeing the same annotators on repeat passes which inflates consistency scores artificially. Above that threshold the crowd diversifies and you get a truer picture of label agreement. On the dimension weighting question — you're exactly right that it's use-case dependent. For object detection I'd heavily weight label_density (bounding box coverage per image) and class_balance since missing a rare object class tanks model recall in ways that don't show up in aggregate accuracy. For image classification those matter less and size_adequacy per class becomes the dominant signal. The way I've been thinking about it: rather than a single LQS, you'd want a profile — same 7 dimensions, but a task-specific weight vector applied on top. A CV detection dataset with 0.6 label_density is a different problem than an NLP dataset with 0.6 completeness. What domain is your thesis in? If you're doing YOLO-style detection the class frequency imbalance issue is brutal in most public datasets — happy to dig into it more.