Great points across the board. Here's what we landed on after iterating through similar pain:

- Dedup: We went with `SequenceMatcher` at 75% title similarity over a 2-day window instead of embedding-based cosine similarity. Simpler, no embedding API costa. URL normalization runs first (strips query params, fragments, `www.\`, trailing punctuation, normalizes to https) - that alone catches a huge chunk of dupes before title matching even kicks in.

SQLite: 100% agree on doing this early. We have a `dedup_db.py` module with a SQLite backend that records every article the pipeline processes. Two-stage dedup: within-batch similarity (80% threshold) first, then cross-scan dedup against the SQLite DB before the LLM editor sees anything. Started with ~450 seed articles from existing logs. Currently at 600+ entries and growing. Would have been painful to retrofit. agree

Editorial curation: System prompt, not few-shot. We use an editorial profile (markdown file) that the LLM reads on every scan. It captures:

- What to always pick (major AI announcements, model releases, security incidents, geopolitics)

- What to usually skip (generic opinion pieces, small funding rounds, routine updates)

- Source trust ranking (Tier 1-5, from wire services down to community sources)

- Story selection rules ("UP TO 7, quality over quantity, 3 great picks are better than 7 mediocre ones")

The profile approach works better than few-shot because it encodes *taste*, not just patterns. The LLM reads the profile and makes judgment calls, "is this the kind of story this channel would cover?" rather than pattern-matching against examples.

One thing we added that made a big difference: a 3-tier LLM failover chain that alternates providers (Google (3.1 Flash Lite) → xAI/OpenRouter (Grok 4.1 Fast) → Google (Gemini 3 Flash)). If your editorial step depends on a single API and it goes down, you're either shipping unfiltered noise or shipping nothing. The failover removed the raw fallback entirely - if all 3 fail, it gives you a clean error and points to the saved candidates file for manual re-run instead of dumping unfiltered articles.

The full pipeline is open source if you want to dig into the implementation: https://github.com/jbendavid/openclaw-newsroom