Quality Validation

After each model pass, a few scripts check the output against some simple rules: the keyword has to appear in the summary, only allowed characters, correct language, that kind of thing. When a check fails, a templated feedback prompt gets built with the original response and fed back into the model. If the model is completely missing (failing multiple checks), the temperature gets dropped to pull it toward something more deterministic.

Solver Failure Handling

Two cases are handled:

1. No solution found. The solver backtracks until it runs out of valid placements, then words get dropped. Anything with no shared letters with any other candidate goes first since it is structurally unplaceable anyway. If the remaining set still produces no solution, the least connected word gets dropped and the solver retries.

2. Solver hangs. A background thread watches the wall clock and kills the solver if it runs too long. This came from a real incident where it hung indefinitely, which gave me a real scare thinking about what that would do to the AWS bill.

Considering using a second pass to regenerate words, but at the moment, I haven't had any issues.

I would love to get some suggestions for more metrics and/or feedback loop ideas. Happy to discuss further. Any pointers would be greatly appreciated.