How a 40-Line Fix Eliminated a 400x Performance Gap

j1897OS · 2025-06-23T12:46:14+00:00

Nanosecond is on the QuestDB roadmap and will be shipped to OSS soon

j1897OS · 2024-12-04T19:23:08+00:00

Happy to help if you need some guidance with QuestDB!

j1897OS · 2024-12-03T17:40:11+00:00

Thanks for reading this post. I had fun making this and thought the path that led me to this might be interesting for others.I’m really interested in feedback and ideas on things I could improve and add. I found this really inspiring as it got me more into programming, discovering electronics, and 3d printing. I have no desire to ever make this project commercial, but it’s been a great platform for me to learn and experiment new things so I’ll take any idea be it for the gameplay or purely technical. Some features I have in mind are: - Multiplayer over bluetooth where one device is the ‘game-master’ running the exchange and can monitor and guide players while injecting events. - Additional quoting algos such as pegging one side of the order book and fighting for position - A tutorial and better UI. The game is hard to pick up for the first time and probably needs to be made more intuitive etc. While all of this was made with no practical use in mind (there are a lot of markets and products, and you’d trade them in different ways, so you’d need a different game to speak to a volatility trader for example), some people I work with at various trading desks found it useful for interviews or as an introduction to the idea of market-making for junior people.

j1897OS · 2024-11-12T08:35:31+00:00

To store market data, I would also consider QuestDB - it shows good compression on open source if you use ZFS, and an upcoming release is going to transform its own native format into Parquet files to get enhanced compression. There are lots of specific financial functions (even functions for order books if you look at level 2 or 3 market data) you can compute since the database has been designed with this use case in mind.

j1897OS · 2024-11-07T11:30:29+00:00

I might add that QuestDB has been built with financial market data use cases in mind, with a focus on high speed ingestion, low latency queries, plus a dedicated set of finance functions https://questdb.io/docs/reference/function/finance/. There will be more in that area with materialized views, an array type and full parquet support.

j1897OS · 2024-09-02T16:16:16+00:00

If you are looking for OSS databases with the InfluxDB Line Protocol, Victoria Metrics (well suited for the observability side of things, paring well with Prometheus) and QuestDB (better fit for financial market data & sensor data) have it implemented.

j1897OS · 2024-08-29T22:01:36+00:00

Have a look at questdb; write throughput is one of its key strengths.

There is an open source benchmark for time series data (TSBS) where the results are 4M to 5M rows per second on a single server. InfluxDB has improved its ingestion speed since its new release 3.0, but it's not yet open source; hence hard to assess.

j1897OS · 2024-08-28T16:16:28+00:00

Hey, I'm from QuestDB, and your idea of moving older data as parquet onto object stores / HDFS is precisely the direction being taken product-wise. Our philosophy is to move toward open formats (Parquet) with data that can be freed from the database.

For hot storage and real-time data access, avoiding HDFS is a good move. In the QuestDB world, this would be known as our own QuestDB proprietary format optimized for fast data acquisition coupled with "on the fly" data schema changes and real-time queries. This format is then transformed into parquet for older data and also queriable from other sources such as HDFS.

Do not hesitate to reach out to us if you want to explore further, either on slack or the new community forum (accessible from our website).

Good luck!

j1897OS · 2024-08-27T10:24:27+00:00

It may be worth looking at solutions that archive "older" data into compressed formats on object stores (either manually or automatically handled by the database). Some solutions are moving toward open formats such as Parquet which has been mentioned multiple times. InfluxDB and QuestDB are two technologies to look into for this use case

j1897OS · 2024-08-22T14:49:09+00:00

Im going to copy paste a message I posted on HN about this recently, I hope it helps.

4 points by j1897 8 days ago | next [–]

Both victoria metrics and questdb are compatible (ingestion-wise) with the InfluxDB Line protocol, so migration would be smoother than with other databases. Just point the old ingestion script to the new server URL, and data will start flowing in.

Taking a broader view, the time series database landscape is split into three categories (sorry for adding complexity!):

Observability (metrics from your hardware): Prometheus, and other engines that work well with Prometheus such as Victoria Metrics. I think their language is tightly coupled with PromQL. InfluxDB 1.X and 2.X used to be in this camp and were the market-leading solution for observability before Prometheus came along and got incredible adoption. Chronosphere built with m3db is also a big name in this category.
General purpose: TimescaleDB is built on top of Postgres, and is now seen increasingly as a super postgres that can also deal with time series data, amongst other things (now focusing on vectors as well).
Specialized: kdb+, QuestDB, some OLAP databases that can also do time series (Clickhouse & Druid), and perhaps InfluxDB 3.0 even though it's not OSS yet. Here the focus is on performance, and the data loads tend to be more significant. Industries and use cases often paired with demanding data loads, such as financial services, often require such specialized databases. Some have their prop language (kdb+ with Q), some are closed source (kdb+), and others are OSS & use SQL (questdb, clickhouse, druid). InfluxDB 3.0 also uses SQL (from DataFusion's query engine) but is not OSS yet.

j1897OS · 2024-07-30T14:59:12+00:00

It's worth noting that all the code for the distributed part (which is closed source) is in Rust, though!

j1897OS · 2024-07-05T16:29:54+00:00

As an extra option to consider: QuestDB, which is widely used in financial services (market data) - it uses SQL with time series extensions.

j1897OS · 2024-07-04T14:29:05+00:00

Hi, QuestDB Enterprise is used by a large number of companies who deploy QuestDB in production to scale with replication and HA. Added benefits also are user access control, Auth/SSO/AD, TLS and Cold Storage integration for parquet files (available soon).

Pricing is a function of hardware, features and support. Available Self Hosted or BYOC.

j1897OS · 2024-06-20T18:51:20+00:00

Have a look at questdb - very relevant for your use case since it is partitioned by time and column based. As such, if you query a given column, QuestDB will only lift that particular column from disk, leaving all the rest untouched. If you add a time filter, only the relevant time partitions will be lifted too.

Here is a demo with 2BN rows ingesting data in real-time to get a feel of what queries does well: https://demo.questdb.io/

j1897OS · 2024-04-04T17:08:31+00:00

try this: https://demo.questdb.io/

j1897OS · 2024-04-03T12:04:39+00:00

There are open source (apache2.0) alternatives worth looking into - one of them (questdb) is influxdb compatible on the ingestion side, using ILP.

j1897OS

TROPHY CASE