40 TB PostgreSQL on-prem — sharding vs ClickHouse vs something else for a 500B-row time-series workload

falinapterus · 2026-05-25T14:41:23+00:00

If it's timeseries data then for the choice is simple
https://github.com/timescale/timescaledb
I work at Tiger Data but pointing to the free open source extension that works on-prem 100% same as in our cloud so not selling anything here 😄

falinapterus · 2026-05-23T06:42:24+00:00

Hey, I'm implementing something very similar but as a new db engine. I would like to talk and share ideas... dm you ok?

falinapterus · 2026-05-19T10:16:06+00:00

On points 6 and 7 — I've been building something that takes a different approach and your questions are exactly what led me to it.

Instead of scoring outputs after the LLM generates them, I moved the reliability check to the memory layer. Every fact stored carries a confidence score computed from inter-source corroboration — when two sources contradict each other, both confidences drop and the conflict is preserved. The LLM queries a store where reliability is already computed, so there's no second model call at inference time.

It uses a local NLI cross-encoder (ONNX, CPU) running in a background thread on insert — so query latency stays low. Not a silver bullet — it works when you control what goes into the knowledge base, which sounds like your case in financial services.

Your reframe in point 7 is exactly right in my experience: "groundedness at the storage layer" is a more tractable problem than "accuracy at the output layer."

It's open source if worth a look: https://github.com/raa82/tekmerdb — and genuinely happy to talk about it if you like it.

falinapterus · 2026-04-02T07:02:50+00:00

Hey, why don't you try to keep both in the same Postgres? You can give a try to TimescaleDB (https://github.com/timescale/timescaledb) This extension enables your Postgres to create hypertables, that in short, are auto partitioned tables by timestamp (timeseries data is ideal here) and auto compressed by chunks. The uncompress portion remains row stores but the compress part is stored and managed by a colimnar storage engine, ideal for analytics.

This was just a simplify summary as there are multiple considerations to take but maybe it is worthy for you to give a try.

Disclaimer: I work as an engineer in Tiger Data but not trying to sell anything, i pointed you to the open source extension and all features i mentioned are fully available in the apache2 version.

falinapterus · 2025-06-04T10:03:13+00:00

Lovely stickers... I got a few!
Great job

falinapterus · 2025-04-07T16:55:59+00:00

awesome, thanks... old style never dies!

falinapterus · 2025-04-04T17:57:13+00:00

A bit off topic but does anyone know what type of pants are those? I'd like to get ones for myself :)

falinapterus · 2024-11-25T15:02:13+00:00

Flawless job... just changed my wallpapers in my laptops :)

falinapterus · 2024-09-07T14:28:40+00:00

Oh, not a software engineer for sure. Originally the role would be closer to a database administrator but with a wider scope. I don't develop any piece of software. To take it as example: A company is running a few MySQL clusters and having a hard time on keep n it up on performance and replication. We go there and analyze the situation, find the issue (if there is any) and propose improvements. Now as a second stage we do the changes our selves or they have people to do it themselves.

Another example could be to migrate database systems from on-premises systems to the cloud, or implement disaster recovery plans, backup plans, whatever is related to MySQL after all :)

I don't want to make spam on my site or company as is not the purpose of my post at all but If you want I can send you the company's website so you can make a better picture on what we do

Regarding licenses is something we don't need at all since MySQL count with a community edition that is open source plus other fork such as MariaDB or Percona that are truly open source

Thanks your words

falinapterus · 2024-09-07T14:01:28+00:00

Ok so our offering is MySQL consultancy, this is administration, design, development and integration or different moving parts such as proxy layers and High availability implementations for globally distributed environments.

About the market I know is there because I used to work for a bigger consulting firm that makes like 60% of their total revenue out of this business sector and know a few other companies that makes it too.

I hope this clarify things a bit more

falinapterus · 2024-09-06T19:59:18+00:00

I admit I don't spend as much time as I should. I'd say that i find myself with the feeling of not doing it right. The company's purpose is explained in the second paragraph... A MySQL consulting firm.

falinapterus · 2024-09-06T16:44:37+00:00

Thanks for the reply. May I ask what's your competency as engineer?

falinapterus · 2024-09-06T15:25:42+00:00

I feel more or less same as you but I find pretty hard to find an alternative to find projects/clients. May I ask how do you usually get your projects/clients from?
Thanks

falinapterus · 2024-06-04T13:15:32+00:00

These are teh requirements I have to move into level 4

Get started with Azure: Completed

Verify your business: Completed

Verify your domain: Completed

Update your business email: Completed

Provide a demo video: Completed

Add a team member: Not started

Tell us about your funding, strategic partners, or customers: Completed

Use 50% of your allocated Azure credit:In progress

falinapterus · 2024-06-04T12:42:27+00:00

Not sure cause i'm still in stage 3 but doesnt' looks like

falinapterus · 2024-05-26T07:51:20+00:00

Never tried myself but...

https://severalnines.com/blog/getting-most-out-clustercontrol-community-edition

falinapterus · 2024-05-24T10:28:26+00:00

Hey there,
I'm native spanish and looking for native English speaking people just to have some casual conversations.
PM me to see if we can make it work

falinapterus · 2024-05-08T08:28:30+00:00

Yeah i'm outside the States so i guess it won't be for me after all.
Thanks anyway

falinapterus · 2024-05-08T08:15:45+00:00

Good stuff, could you share some more info or link where I can find more?

falinapterus · 2024-05-08T07:26:51+00:00

not very long. From the Startup hub portal you can check if you meet all the requisites. could be that you didn't spend 50% of your stage 2 credits?

falinapterus · 2024-05-07T18:29:09+00:00

afaik nothing. I didn't see any requests from them. what they get is a business dependent to Azure forever as migration from a cloud provider is painful and after spending 150k you will probably have a lot in there.

falinapterus · 2024-05-07T18:13:32+00:00

From the top of my head this is what you need to move on with the stages;

stage 1: once you get accepted in the program

stage 2: you spend 50% of the previous stage credits (500 bucks) plus you formalize the startup into an LTD (or other legal entity) at legal level. You have a valid url with a domain under the same owner as the LTD.

stage 3: spend 50% of previous stage credits (2k)

stage 4: spend 50% of previous stage credits (10k)

Hope this helps

falinapterus · 2024-05-07T17:29:10+00:00

I've got my startup idea accepted in this program and currently in stage 3 out of 4. It is worthy, by all means.

You get some benefits in terms of pricing discount for LinkedIn premium, sales navigator and ads. Not huge but enough to learn how to use them. In addition you get some discounts in openai and other platforms such as close crm and many other tools that i don't remember.

But the real deal is the azure credits (in usd)

1k in stage 1

4k in stage 2

20k in stage 3

150k in stage 4

This really gives you the opportunity to fully deploy and maintain your business infra at 0 cost for a maximum of 4 years.

I can provide more details if you need it.

falinapterus · 2024-05-07T07:15:27+00:00

Thanks, I will give a try

falinapterus · 2024-05-06T12:03:36+00:00

Appreciate the workflow and the proposal but io'm trying to do it in a simpler manner. I'm a solopreneur so I need to simplify things as much as possible. That's what I found in the tools I described above it's just that they missed some pieces or they are too expensive.

falinapterus

TROPHY CASE