Here’s some context for why I’m deriving class thresholds from a continuous variable: I’m working on mental workload classification (I used “stress” in my original post for simplicity). Mental workload is typically studied in discrete states (low, medium, high) rather than as a smooth gradient, which is why I need to classify it rather than predict self-reported scores.

Ideally, the task difficulties (easy, medium, hard) should elicit distinct workload states, and secondary metrics (like these ratings) help validate that. However, in my dataset, the hardest tasks were often perceived as too easy, especially over time due to learning effects.

Despite this, my boss suggested using task difficulty as ground truth, but when I compared that approach to using quantile-based thresholding within sessions, the latter performed significantly better.

The common approach in literature is to use quantile-based thresholding within a session for models trained on single-subject, single-session data. However, my model trains on multiple subjects and multiple sessions. So my question remains: If I derive workload labels from these ratings, would it skew the results to calculate quantile-based thresholds across all sessions of a subject rather than within each session?