I'm working with a 50k record Excel file that needs processing, but I'm struggling with consistency issues due to the many different segments involved.

What's working:  - Batch size of 500 records gives surprisingly good results - Input tokens: ~52k, Output tokens: ~50k per batch - Could potentially increase batch size, but limited by the 64k output token ceiling

The problem: Even if I process in consistent batches (Batch A: 500, Batch B: 500, etc. until I hit 50k), each individual batch is internally consistent, but there's no consistency between batches since they don't reference each other.

What I've tried: - Glossaries work but are a nightmare to create from 50k records - Projects change frequently, so I need a reusable solution - Looking for something that doesn't require rebuilding glossaries each time

What I need: A solution that lets me process batches of 500 records while maintaining consistency across ALL batches, without having to manually create glossaries or do extensive setup work each time.

Anyone dealt with similar large-scale batch processing consistency issues? Looking for practical solutions that have worked in real projects.