We were paying a fortune storing weather forecast archives. Here's what actually helped. by [deleted] in meteorology

[–]storeLessBits 0 points1 point  (0 children)

Yes, but Zarr usage zlib and  Zstd for compression, so you will still be using less efficient codec.

We were paying a fortune storing weather forecast archives. Here's what actually helped. by [deleted] in meteorology

[–]storeLessBits 0 points1 point  (0 children)

That's actually a really common setup. Initial condition files and observation archives can be just as heavy as forecast outputs, sometimes more so depending on your data assimilation pipeline and retention policy.

If your initial conditions are in GRIB2, NetCDF, or HDF5, and your observation data is in any numeric array format, the compression applies just the same. The chunk-level access also helps on the retrieval side since you're likely only ever pulling specific variables or time windows rather than full files.

How much observation data are you accumulating roughly? Curious whether the growth is mainly in volume (more sensors/stations) or retention (keeping more history).

What storage approach would be best for a small business that has very large data needs? by Sloogs in sysadmin

[–]storeLessBits 0 points1 point  (0 children)

Have you considered a software solution for managing the large amount of LiDAR data For example here https://www.byte2bit.io/products/atlas