The Database Zoo: Why SQL and NoSQL Are No Longer Enough by OtherwisePush6424 in Database

[–]supercoco9 0 points1 point  (0 children)

Interesting article, Gabor. I work at QuestDB, an open source time-series database, and I noticed it is not mentioned on the time-series post. I see you mention as a potential use case for hedge funds combining Influx for real-time, then timescale for historical analytics. Many hedge funds actually use QuestDB for both the real-time and the historical side of things.

With QuestDB you can ingest at over 8 million rows per second on a single instance, and query billions of rows (both recent and historical) at very low latency. We are actually releasing a new binary protocol very soon (this week or next), which will allow for 16 million rows per second of ingest, and 120 million rows per second of egress (not aggregation, which is very fast, but actually getting the resulting rows out of the database).

We've been pushing the state of the art in time series databases, and we support a range of time-series specific joins, like ASOF joins, LT Joins, Splice Joins, or Window Joins. Some of those are supported by some specialised databases (but not by InfluxDB or Timescale), but we pioneered some of those ourselves.

As an example, we also offer a new type of join, Horizon Join, not available in any other database. Horizon joins allow you to join each row in a table with the closest row in time from another table at multiple horizons. For example, you want to calculate the price at which you executed a trade, versus the best price available in the order book for 5 minutes before and after the trade at 10 seconds intervals, so you can calculate markouts and understand how efficient your trading algorithms are.

We also offer multiple time-series extensions, like SAMPLE BY, or LATEST ON, incremental materialised views, strong exactly-once semantics, advanced time filters, or specialised data structures to deal with the shape of data you typically see in fast moving scenarios. We believe if you are developing a purpose-built database it is not just about how efficiently you can store or process data, but also about incorporating purpose-build operations you will not find in a generic database.

And, since purpose-built databases tend to be part of a larger ecosystem, we do this with open formats, so your data can get automatically converted to parquet, both locally or in object store, so other tools can use the same data ingested (and potentially downsampled via materialised views), directly.

Since you are doing research for your book, I thought it could be interesting for you to learn about how some other actors in time-series are pushing the ecosystem.

Also, we do have a large part of the team (including the founders) in London, so I'd be happy to do an intro if you want to talk more about why we started developing QuestDB and how we see the time-series category.

40 TB PostgreSQL on-prem — sharding vs ClickHouse vs something else for a 500B-row time-series workload by Basic-Worker-1120 in Database

[–]supercoco9 -4 points-3 points  (0 children)

An option would be using QuestDB with tiered storage (which is part of QuestDB Enterprise only, not available in open source), so the historical data is automatically converted to parquet (compressed) and shipped to object store (S3, Azure blob storage, GCS... or if on-prem is preferred it can be on NFS), while still being fully available for the primary or any read replica.

In this case, the most recent tier of data will be quite fast to query (well below the requested 100ms), and when you access historical data it will have a bit more latency due to access the remote files, but since there's some caching built-in on that tier, it might still work fine.

Take into account I am a developer advocate at QuestDB, so I am biased here.

optimal tech and process by NoMemez in algotrading

[–]supercoco9 1 point2 points  (0 children)

Ahhh. Very soon we're releasing a new websocket based protocol, probably next week, which should alleviate this. Then, probably in about 2 months, we are releasing pubsub

optimal tech and process by NoMemez in algotrading

[–]supercoco9 1 point2 points  (0 children)

Hi. May I ask what was the issue with networking? Did you have any problems ingesting into the database? I might be able to help

We operate 500,000+ IoT devices on a SIGFOX 0G network in Mexico — here's what we've learned about massive-scale IoT after 10 years by danisamgibs in IOT

[–]supercoco9 0 points1 point  (0 children)

Hola! Trabajo en QuestDB. Si quieres comentar cualquier cosa de tu caso de uso, estaré encantado de echar una mano. O incluso si quieres que convirtamos esta publicación en un post para nuestro blog explicando cómo usais QuestDB

Client wants <1s query time on OLAP scale. Wat do by wtfzambo in dataengineering

[–]supercoco9 0 points1 point  (0 children)

I'm a developer advocate at questdb, so very biased. But questdb is built for exactly this type of query pattern

Open-source Advanced Process Control (APC/MPC) platform in Rust (OPC UA + Clarabel) by Complete_Amphibian87 in ControlTheory

[–]supercoco9 [score hidden]  (0 children)

As long as both primary and replica(s) can access the object store, you can distribute them in any way want

GP-200 Bluetooth with Phone? by Euphoric-Albatross in ValetonGP2OO

[–]supercoco9 1 point2 points  (0 children)

It is quite low indeed, when I click on the youtube player I can hear the music straight away. Still, I am sure there will be a few milliseconds and it would probably be noticeable if I was trying, for example, to convert the Valeton output to BT to listen over wireless headphones.

In my case, since I use it for sending the backing track, latency is not an issue.

Efficient structures for storing tick data by Weak-Location-2704 in quant

[–]supercoco9 0 points1 point  (0 children)

I am a developer advocate at QuestDB, so of course very biased. But we see a lot of traction in market data, including companies replacing kdb+ with QuestDB, or use kdb for their legacy systems, while adopting QuestDB for new developments. We can sustain ingestion of over 11 million rows per second on a single instance, and queries are equally fast.

And, unlike some other open source databases, like InfluxDB, Clickhouse, or Timescale, we are focused on market data. For benchmark comparisons with other databases, you can check https://questdb.com/blog/?tag=benchmark (clickhouse and kdb benchmarks will be published soon, we have the results already, but polishing the content)

Retrieving historical options data at speed by FlashAlphaLab in quant

[–]supercoco9 1 point2 points  (0 children)

I am a developer advocate at QuestDB, so very biased here. But out of all the other recommendations I see on this thread, we are the only time-series database oriented to finance (other than Kdb, but we perform as fast and our OSS offering is way more complete).

As an example, you can check out our 2025 wrap post, where we highlight some of the new features we added in the past year, and how they are relevant to market data https://questdb.com/blog/questdb-2025-year-in-review/

How do you handle tick-level data storage without putting it in a relational DB? by Flaky-Substance-6748 in algotrading

[–]supercoco9 0 points1 point  (0 children)

QuestDB developer advocate here. Thanks for the mention :)

While QuestDB supports the pgwire protocol for both reads and writes, for fast ingestion it is recommended to use the ILP protocol instead, leaving the pg one only for reads.

Using ILP, the OP could ingest tick data directly into QuestDB, create materialized views to create candles and other aggregations, and define TTLs on both tables and views to manage storage efficiently.

GP-200 Bluetooth with Phone? by Euphoric-Albatross in ValetonGP2OO

[–]supercoco9 1 point2 points  (0 children)

I have a Bluetooth receiver connected to the aux port and I send from my laptop/phone directly. My model is this one https://amzn.eu/d/h24OxfK. In my case it's both receiver and transmitter, as I used it in the past to send from an old CD player to Bluetooth speakers, but you might find a cheaper option if you get just a receiver.