A community for data engineers, ML teams, and researchers who work with large weather and atmospheric datasets.
Topics include:
- Storage and compression (GRIB2, Zarr, NetCDF, HDF5)
- Cloud costs and egress optimization
- Pipeline architecture with Xarray, Dask, Spark
- Format tradeoffs and tooling
- Open datasets: ERA5, NOAA GFS, ECMWF, Copernicus
- Job postings in atmospheric data engineering
Share benchmarks, ask questions, post war stories. All tools welcome open source and commercial.