Case study · A national climate research centre
Shrinking a national climate archive.
How 2.1 PB of model output became 140 TB, with error bounds the scientists signed off
- Research data managers
- Climate scientists
- HPC and storage teams
Download the case study12 pages · PDF · 2026
- smaller archive
- 15×
- smaller archive
- largest error on temperature
- 0.01 K
- largest error on temperature
- faster reads for analysis
- 3×
- faster reads for analysis
What’s inside
The whole story, with the numbers.
The archive
Decades of reanalysis and model runs in NetCDF and GRIB, growing by hundreds of terabytes a year.
Choosing the bounds
How the science team set an error bound for each variable, and which ones stayed lossless.
Proving it
Re-running published analyses on the compressed archive and comparing the results.
Fitting the pipeline
Zarr in object storage, read slice by slice from xarray, with no change to the analysis code.
Contents
6 chapters.
- 01The problem with keeping everything
- 02Setting error bounds with the scientists
- 03Running Compress over the archive
- 04Checking the science still holds
- 05The savings, year by year
- 06What we’d do differently
Free download
Where should we send it?
Fill this in and the download link appears straight away.
It’s yours.
Rather see it on your own data?
Every demo is run by the people who build Compress, on a sample of your own files.