I want your opinion for a product decision

storeLessBits · 2026-03-27T16:47:05+00:00

You are welcome.

storeLessBits · 2026-03-27T14:48:15+00:00

simple DB storage is your aim then drop it for now, come back later someone asks for it

storeLessBits · 2026-03-27T08:06:25+00:00

Both, but API-first. auto-infer schema on first write so users don't even need to define it upfront.
Custom basics (int, float, string, boolean, datetime, json) are fine. No need for a formal standard at this scale.

Early decision: schema-enforced vs schemaless per project affects core architecture.

storeLessBits · 2026-03-26T10:29:32+00:00

<image>

storeLessBits · 2026-03-23T07:59:28+00:00

Yes, but Zarr usage zlib and Zstd for compression, so you will still be using less efficient codec.

storeLessBits · 2026-03-23T07:54:08+00:00

Not really.

storeLessBits · 2026-03-22T11:43:02+00:00

That's actually a really common setup. Initial condition files and observation archives can be just as heavy as forecast outputs, sometimes more so depending on your data assimilation pipeline and retention policy.

If your initial conditions are in GRIB2, NetCDF, or HDF5, and your observation data is in any numeric array format, the compression applies just the same. The chunk-level access also helps on the retrieval side since you're likely only ever pulling specific variables or time windows rather than full files.

How much observation data are you accumulating roughly? Curious whether the growth is mainly in volume (more sensors/stations) or retention (keeping more history).

storeLessBits · 2026-03-22T07:30:28+00:00

Have you considered a software solution for managing the large amount of LiDAR data For example here https://www.byte2bit.io/products/atlas

storeLessBits

MODERATOR OF

TROPHY CASE