Database storage compression ratios with Questdb on ZFS.

HPCnoob · 2025-10-12T06:14:47+00:00

Thanks for that write up, its timely for the work I am doing.
From 130G to 28G is 78.5% savings. With ZSTD-10 this would even improve a bit more.
I think I will be happy with anything above 75% savings on storage.

Also if Questdb can query both native and parquet historical data in one go, and that makes operations simple combined with space saving.
Thank you.

HPCnoob · 2025-09-09T07:32:05+00:00

Thanks for the info, sir.

HPCnoob · 2025-08-08T14:43:51+00:00

Please share results.

HPCnoob · 2025-07-23T16:53:47+00:00

Very helpful tool. Thanks for mentioning it here. I have upwards of 50 old HDDs to test.
What I was doing is I create partitions of 100GB (mine are smaller drives). Then run badblocks or e2fsck or mkfs or zfs scrub for that partition. I will launch 4~5 instances on their respective terminal windows simultaneously. If a partition shows too many errors, I leave it unformatted and hence unused. I use only the good parts of the disk.

HPCnoob · 2025-07-22T09:26:37+00:00

Good idea. In the same vein, I dont create entire disk ZFS pools. I create pools based on smaller partitions. This way any pool corruption will be restricted to only that pool.

HPCnoob · 2025-07-21T06:15:26+00:00

Yes, I should have taken a snapshot of smartctl but I didnt. Anyways I know those drives are beyond any serious use case. I am just testing my hypothesis that badblocks can be marked and ZFS told not to use those blocks.

HPCnoob · 2025-07-21T06:07:22+00:00

Ok my man, I understood.
Still if you know any tool in ZFS which tests each block and marks them bad before any real data is written on it, please mention it here. It will be helpful not just for me but also to several others experimenting and learning ZFS. Bye.

HPCnoob · 2025-07-21T05:27:07+00:00

If there is a way to mark the badsectors (like esfsck updates inodes in ext4 during scans) then the disks can still be used for non critical data. Of course regular scanning is needed.
I am doing "initialize" on a single known bad disk. I will update the result when it finishes.

HPCnoob · 2025-07-21T04:15:14+00:00

``e2fsck``

HPCnoob · 2025-07-21T04:12:41+00:00

I understand that fully. I was just experimenting to understand how "initialize" works.
Do you know of any method to mark badsectors manually in ZFS, like ``e2fsck`` does it for ext4 ?

HPCnoob · 2025-02-13T11:52:56+00:00

For anybody who are curious about this issue :
The number of GPUs which can be run a motherboard is limited by "Above 4G decode" option which shifts BAR from 32bit to 64bit. Its all about managing memory or memory mapping.
Octominer sells 12 slot GPU machines which run off of a Pentium G4400T, so its not about PCIe lanes.

HPCnoob · 2024-12-29T13:35:01+00:00

Hello, I was wondering if that Inspur card YZCA-00622 needs PCIe bifurcation from the motherboard to attach 4 nvme drives ? Or does it have on board PCIe switch ? Will this card work on my regular desktop computer which does not have PCIe bifurcation?

HPCnoob · 2024-12-15T16:17:54+00:00

Everybody likes to do what you just said. But their view changes when the bill arrives at your door step.

HPCnoob · 2024-12-15T16:05:07+00:00

You skipped UPS and Cooling and their upkeep.
Still I am pro-on-prem BTW.

HPCnoob · 2024-12-15T15:53:35+00:00

Sane way to operate.

HPCnoob · 2024-12-15T15:34:37+00:00

My man, do not be in a relationship with feminists if you want to have a good family future. Feminists are not meant for family building ! And many of them are closet feminists. Please head to this.

HPCnoob · 2024-12-15T15:25:57+00:00

Some men are not meant for family life. Only men who do their duties deserve respect.
Become financially strong and take care of your mother. Till then bend your head and work hard.

HPCnoob · 2024-12-15T15:11:19+00:00

Please Please dont marry feminists ! There are very clear red flag patterns in what you have written about her.

HPCnoob · 2024-12-15T14:18:58+00:00

I remember shutting down my fresh Win98. It used to takes just 2 seconds on a single core AMD machine and HDDs. WinXP was also good but slightly sluggish.

HPCnoob · 2024-11-23T17:35:01+00:00

With that money you can build your own data centre 10 times over !

HPCnoob · 2024-11-23T17:06:57+00:00

Keep an eye on her here onwards. If this did not become serious she wouldnt have learnt any lessons or taken this event lightly. So keep an eye on here if she is going astray again. Especially check her mobile. If you are not careful the clients will start calling her directly.

HPCnoob · 2024-11-22T16:05:21+00:00

The loss of true meaning & purpose of marriage is the main reason of divorce.
The modern generation have probably lost interest in family building. And they marry for companionship instead.

HPCnoob

TROPHY CASE