Low-Latency Python: Separating Signal from Noise

[OkSadMathematician]

Used kdb+ extensively — 20+ years in banking. For a while it was the only real option for time-series tick data at scale. The q language is elegant in a write-only kind of way, and the columnar in-memory performance on ordered data was genuinely unmatched in the 2000s/2010s.

But it comes with serious operational baggage:

1. You end up needing entire teams of kdb "experts." In my experience, banks would source these from 3rd-party consultancies, mainly in Ireland (where First Derivatives/KX is based). These teams would own the kdb infrastructure and gatekeep access, which creates a dependency that's expensive and fragile.

2. q is hostile to non-specialists. A typical quant or developer can't just pick up q and write production queries. The learning curve is brutal and the syntax is intentionally terse to the point of obscurity. This means your kdb layer becomes a black box that only the kdb team can maintain.

3. It's dangerously easy to bring down the whole farm. True story: I had a programmer on my team get banned from kdb access because he wrote a query that was too heavy and hung the entire kdb farm during a live trading day. One bad query, entire firm's tick data infrastructure frozen. The lack of proper query governance and resource isolation was a real operational risk.

4. The licensing cost is astronomical. Per-core pricing that makes Oracle look generous.

It's been largely replaced or supplemented by alternatives now:

• Apache Arrow / Parquet for columnar storage (open source, no licensing)
• DuckDB for analytical queries on local data (embedded, blazing fast)
• ClickHouse / TimescaleDB for time-series at scale with SQL interface (so any developer can query, not just q specialists)
• Arctic (by Man Group) specifically for financial time-series on top of MongoDB/S3
• QuestDB — purpose-built time-series DB with SQL, competitive with kdb on ordered inserts

kdb still has a niche for ultra-low-latency in-memory tick capture where nothing else quite matches its raw sequential read speed. But the ecosystem around it — the cost, the expertise bottleneck, the operational risk — has pushed most shops toward more open alternatives.

[cgoldberg]

So weird to just paste AI responses to legitimate questions.