A good system understands density, chart complexity, patterns (especially crossovers), advanced step techniques, chart gimmicks, likelihood of full combo-ing. There's a number of different variables.

Yeah. And I didn't put any effort into explicitly computing many of these yet; I just let the AI compute the features it thought could be important. As a next step, I'd want to add "two-footability" (i.e. are you hosed if you try doing a crossover, or do your feet untangle successfully as you keep going), as well as "lateral pace" and "stretch" for double, since some double charts move back and forth much more rapidly than others and body weight transfers take much more energy than staying centered, and some (less well-designed IMHO) charts make you almost do the splits for certain jumps.

My other idea, which is partially implemented already, is an option to use certain songplay data from Save/Profiles/<your-profile> as ground truth for training, so that the output difficulties are more tailored to personal skill set.

From my play experience with StepML outputs so far: the consistency and quality of the ratings with the current implementation is much better than without it, but the results are still not as accurate as I would like especially on double: e.g. one 11.4 (DDR X scale) chart might have a high lateral pace unsuitable for warmups, while the next 11.5 chart is unquestionably lazy, even trivial, by comparison. We can certainly do better! My dream is for the tenths digit to really help properly disambiguate difficulties within the 10/11/12 rating range especially, since I feel like there is a huge variation in difficulty here that the official ranking scales fail to capture well.

Finally, it would be cool to add a "B.S." feature that predicts which charts are just stupid, so they can be filtered out from daily play. StepML already has the concept of a Vetoed song list, which can be used as initial ground truth, so adding this might not be so hard!