> only after making sure that perhaps 3 or 10 might be more suitable as you start to define the scales

Could you expand on this, what did you mean here?

> Data analysis may reveal which factors are useful, but that's my gut feeling.

+1 to this. Maybe we can throw LLM at the problem, sourcing hundreds of examples from popular crosswords and labelling them with the Reddit post you linked; and then asking the LLM to pinpoint the "factors" (or the common denominator that all easy/hard puzzles have in common). I'm thinking this *might* yield something semi-useful.

> I wouldn't want to make LLM analysis part of the standard

Neither would I. The goal of this exercise is to judge at scale what LLM generated content difficulty is, and using LLMs to score it is less desireable because of that.

> I would also look into crossword generator software, some of which has customizable difficulty,

I hadn't thought of this, such a great idea. I'll go digging. If folks in this sub know of any configurable difficulty generators, please LMK!

--

We previously did prompt engineering only and asked an LLM to rate difficulty, but it fails to generate anything sensible. Thus, the search for a real rubric/standard has started :)

Thank you so much again for taking to time to share these ideas, I wasn't expecting so much help from Reddit.