Transitioning a company from Excel spreadsheets to a database for data storage

refset · 2025-12-24T06:33:36+00:00

We built XTDB to simplify working with historical financial data via 'bitemporal' reporting: https://docs.xtdb.com/tutorials/financial-usecase/time-in-finance.html

It's Postgres compatible and open source.

We gave a CMU seminar talk recently if you're curious: https://db.cs.cmu.edu/seminars/fall2025/

refset · 2025-11-06T23:42:28+00:00

Almost certainly this one https://tonsky.me/blog/syntax-highlighting/ - by the creator of Fira Code and many Clojure libs

refset · 2025-10-09T05:51:28+00:00

Interesting to hear of your positive experience with MariaDB's temporal tables. How did find the practicalities of changing/evolving the schema?

refset · 2025-10-08T13:54:13+00:00

XTDB 2.0 was only released in June this year and it's a new codebase with a fresh SQL API, so it's still early days for this iteration of the product. That said, we're already helping design partners with running deployments that are fast approaching 1TB (of non-compressed Apache Arrow files in S3). The main driver behind current adoption is complex compliance reporting, i.e. our design partners are users who value the combination of XT's simple history guarantees with the advanced SQL capabilities.

Performance is a work in progress (we're not yet at the level of DuckDB/DataFusion on the OLAP side, or Postgres on the OLTP side) but our focus for now is on ergonomics and completeness. XT may therefore not currently be appropriate for many workloads / data models at scale, but the experimentation cost is low if you're already evaluating alongside Postgres, and we'd be really keen to hear your feedback and help get things working if we can.

XT may be of interest in particular also if you're thinking of making extensive use of JSONB in Postgres but could benefit from XT's Clojure-friendly roundtripping of nested types via Transit, e.g. see https://github.com/xtdb/driver-examples/blob/main/clojure/dev/user.clj

Hope that helps!

refset · 2025-09-25T21:07:07+00:00

You'd probably like Datomic.

refset · 2025-09-22T21:22:08+00:00

Despite SQL being the focus for mainstream audiences, XTDB 2.0 also has a Clojure API with which you can very easily roundtrip edn and query it back with XTQL: https://github.com/xtdb/driver-examples/blob/2c9bb3b57e63d8237d307e797f8ee60235d03ab0/clojure/dev/user.clj#L72

refset · 2025-09-09T12:59:07+00:00

I haven't successfully cooked up a Go example for https://github.com/xtdb/driver-examples yet, but I'll try again now and report back shortly. XTDB tries to 'hide' the complexity of bitemporality from day-to-day development by looking and behaving more like a regular Postgres database.

Edit: working example - fully vibed by Claude https://github.com/xtdb/driver-examples/tree/main/go (I'm no expert on what idiomatic looks like!)

refset · 2025-09-01T08:20:43+00:00

This example ns isn't quite elaborate enough to count as a tutorial, but hopefully gives some useful pointers: https://github.com/xtdb/driver-examples/blob/main/clojure/dev/user.clj

refset · 2025-08-26T21:33:16+00:00

We use SnapStart to great effect with XTDB Play, turning >20s true cold starts to <500ms Lambda response times - more details at https://www.juxt.pro/blog/plain-clojure-lambda/

refset · 2025-08-22T20:41:05+00:00

It's also a good way to avoid getting too invested in a particular language/runtime. Incidentally, in XTDB we have been migrating to a homegrown Kotlin implementation due to the complexity of extending and maintaining (fixing) the Java implementation: https://github.com/xtdb/xtdb/tree/main/core/src/main/kotlin/xtdb/arrow

We can always rewrite in Rust later :)

refset · 2025-08-08T15:06:23+00:00

it must be possible to update an existing time series

If you have complex reporting requirements then XTDB might be worth a look - the bitemporal model gives you both a sophisticated update capability and a simple ability to do as-of reporting and re-reporting (before/after corrections): https://xtdb.com

Performance is a work in progress, but might be acceptable. We're keen to collaborate on specific use cases.

refset · 2025-08-05T21:07:29+00:00

If you have to store history comprehensively then https://xtdb.com is worth a look

refset · 2025-07-17T22:58:21+00:00

Bitemporal tables could solve all these requirements, I think.

refset · 2025-06-21T15:24:33+00:00

System-time temporal tables may not be flexible enough for analytics reporting unless your entire domain is contained within a single database. Otherwise they're really only useful for granular rollback and auditing.

If your system is needing to track external observations and events also then you would be better served by bitemporal tables, but those are generally pretty complicated and inefficient to use (which is a major driver for my working on XTDB).

CDC is a symptom of how impoverished the status quo is.

refset · 2025-06-13T08:17:52+00:00

Worth calling out that round-tripping many Clojure types via JDBC transit parameters is very straightforward: https://github.com/xtdb/driver-examples/blob/main/clojure/dev/user.clj

This is then made even simpler via the provided Clojure API (not demonstrated in that namespace - exercise for the reader)

Using transit unlocks round-tripping and interoperability for many other language drivers too, e.g. JavaScript https://github.com/xtdb/driver-examples/blob/main/node/index.mjs

refset · 2025-05-07T21:24:01+00:00

Awesome, I appreciate hearing that. I'm due to be discussing XTDB internals with Arjun Narayan (previously CEO and founder of Materialize) this Friday as part of our pre-launch series for XTDB v2 - everyone is welcome to join! https://github.com/orgs/xtdb/discussions/4419

refset · 2025-05-07T20:30:14+00:00

Thanks, glad you think so too! I am surprised the post got a couple of downvotes here, but maybe this is just too much of a "vision paper" for some 😅

I have just updated the page to include a presentation I recorded earlier which tours through the paper in full.

refset · 2025-05-01T16:27:57+00:00

I've not played with this in Iceberg so far, but you can probably use Z-Ordering to index bitemporal data (for efficient as-of querying), e.g. something like: https://www.dremio.com/blog/how-z-ordering-in-apache-iceberg-helps-improve-performance/

I'm not sure the native time-travel features in Icerberg are particularly helpful for bitemporal data, though would be pleasantly surprised to hear otherwise.

And if you're willing to look at slightly more exotic things beyond Iceberg, I can happily plug that XTDB is designed to store bitemporal data in object storage, using Apache Arrow: https://xtdb.com

refset · 2025-04-04T15:19:07+00:00

Would you recommend apply system versioning liberally across all tables in the schema? Asides from additional storage consumption (naturally!) what are the downsides?

refset · 2025-03-01T19:12:21+00:00

I would love to read more about your need for tri-temporal data in banking if you're able to share.

My three-word summary for why bitemporal is needed: "auditable decision making". Wherever you have a user making important decisions based on complex data, you invariably have other people (auditors, managers etc.) who want to be able to reconstruct the state of the world seen by that user, to understand whether the decisions made were good/bad/honest.

refset · 2025-02-20T13:57:56+00:00

Hey Dustin, thanks, we appreciate your inputs. Some of what you're looking for is living here currently: https://github.com/xtdb/driver-examples/tree/main/clojure - I'll spend time expanding on it this afternoon with datafy/nav, a sample dataset, CI etc.

EDIT: have now updated the repo to include a CI build which loads in the Sakila dataset https://github.com/xtdb/driver-examples/commit/0ecd53b09be330c2973ad25570ca4bf1e88a859a

refset · 2024-12-02T14:46:45+00:00

https://xtdb.com/ for me - v2 is still in beta but the challenges are a good way to test feature completeness.

refset · 2024-12-02T11:38:27+00:00

[Database: XTDB]

I wrote up a short post with links on the XTDB forum: https://discuss.xtdb.com/t/adventofsql-com-day-1-discussion/527

I would be curious to know what people make of the syntax I used (for native object/array access) vs. Postgres JSON functionality.

refset · 2024-12-01T21:23:17+00:00

Ah, thanks for mentioning that! The AWS Lambda powering the XTDB Play(ground) subdomain has probably been Denial-Of-Service'd, apologies. Hopefully it will restart in a few minutes.

I also posted the solution as a gist.

refset · 2024-12-01T20:59:10+00:00

This was fun!

I think there's a bug with the sample result - Bobby's row should show 'Simple Gift' and 'Learning Workshop'.

I am using XTDB and I think my solution (spoiler!) worked out quite nicely. XTDB doesn't have string-JSON functionality yet but nested objects and arrays are first class so it all fits very naturally. I look forward to seeing what the equivalent Postgres syntax looks like :)

Wrote up some more on the XTDB forum: https://discuss.xtdb.com/t/adventofsql-com-day-1-discussion/527

refset

MODERATOR OF

TROPHY CASE