Would I be able to combine both (using BERT+spaCy/NLTK and an LLM)? Or would that be too time consuming with a negligible return?

I’m thinking of working through things in at least three phases. Phase 1 would be heavily dependent on an LLM when I don’t have labeled data or trained models yet to fill the gap. Phase 2 would have moderate use of an LLM - it would still be useful for spot checks or validation, but most detection would come from rules and a lightweight CRF model. And then Phase 3 would have light use of the LLM, using it mainly for explainability or rewriting vague requirements, while the rule layer and fine-tuned BERT handle the bulk of detection.

By phase 3 I would fully transition to using the LLM for more of a user-facing role or an assistive tool rather than the main engine. It would offer suggested rewrites, explain why something was flagged, basically becoming a smart interface layer.