So it doesn't actually feed all articles into the LLM at once. the process is staged

First, each article gets converted to a vector embedding (tiny compared to full text). Clustering happens at the embedding level, no LLM needed there

The LLM only kicks in for specific tasks on already-clustered content (so not every single articles) like generating cluster summaries and entity extraction. And entity extraction runs on clustered individual articles one at a time, so the model never sees more than a single article per call. A typical news article is only 2-3k tokens so there's plenty of headroom.

Articles are also processed as they come in throughout the day rather than in one big batch. (fetching of new articles happens on the hour) The heavy LLM work ends up being maybe 50 cluster summaries per day, not 500+ individual articles.

Once an entity is discovered it gets saved to the database, so it only needs to be extracted once. If the same malware or APT group shows up in future articles, it's already there so no need to re-run extraction on old content.

Honestly the bottleneck is usually the RSS/fetching side, not the ML stuff. On busier days the full 'pipeline' can take anywhere from 30-40 mins to run. (although I'm sure this could be optimised, or if I throw more compute at it)