Analysis-Ready, Cloud Optimized ERA5

Recipes for reproducing Analysis-Ready & Cloud Optimized (ARCO) ERA5 datasets.

Introduction • Overview • Analysis Ready Data • Raw Cloud Optimized Data • Project roadmap • How to reproduce • FAQs • How to cite this work • License

Introduction

Our goal is to make a global history of the climate highly accessible in the cloud. To that end, we present a curated copy of the ERA5 corpus in Google Cloud Public Datasets.

<details> <summary>What is ERA5?</summary>

ERA5 is the fifth generation of ECMWF's Atmospheric Reanalysis. It spans atmospheric, land, and ocean variables. ERA5 is an hourly dataset with global coverage at 30km resolution (~0.28° x 0.28°), ranging from 1979 to the present. The total ERA5 dataset is about 5 petabytes in size.

Check out ECMWF's documentation on ERA5 for more.

</details> <details> <summary>What is a reanalysis?</summary>

A reanalysis is the "most complete picture currently possible of past weather and climate." Reanalyses are created from assimilation of a wide range of data sources via numerical weather prediction (NWP) models.

Read ECMWF's introduction to reanalysis for more.

</details>

So far, we have ingested meteorologically valuable variables for the land and atmosphere. From this, we have produced a cloud-optimized version of ERA5, in which we have converted grib data to Zarr with no other modifications. In addition, we have created "analysis-ready" versions on regular lat-lon grids, oriented towards common research & ML workflows.

This two-pronged approach for the data serves different user needs. Some researchers need full control over the interpolation of data for their analysis. Most will want a batteries-included dataset, where standard pre-processing and chunk optimization is already applied. In general, we ensure that every step in this pipeline is open and reproducible, to provide transparency in the provenance of all data.

Overview

Location	Type	Description
$BUCKET/ar/	Analysis Ready	An ML-ready, unified (surface & atmospheric) version of the data in Zarr.
$BUCKET/co/	Cloud Optimized	A port of gaussian-gridded ERA5 data to Zarr.
$BUCKET/raw/	Raw Data	All raw grib & NetCDF data.

Files are updated from ECMWF on a monthly cadence (on roughly the 9th of each month) with a 3 month delay, which avoids including preliminary versions of ERA5. The date of the latest available data can be found by inspecting the "time" axis of each Zarr store.

Analysis Ready Data

These datasets have been regridded to a uniform 0.25° equiangular horizontal resolution to facilitate downstream analyses, e.g., with WeatherBench2.

0.25° Pressure and Surface Level Data

This dataset contains most pressure-level fields and all surface-level field regridded to a uniform 0.25° resolution. It is a superset of the data used to train GraphCast and NeuralGCM.

import xarray

ar_full_37_1h = xarray.open_zarr(
    'gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3',
    chunks=None,
    storage_options=dict(token='anon'),
)

• Times: 00/to/23
• Levels: 1/2/3/5/7/10/20/30/50/70/100/125/150/175/200/225/250/300/350/400/450/500/550/600/650/700/750/775/800/825/850/875/900/925/950/975/1000
• Grid: equiangular lat-lon
• Size: 2.05 PB
• Chunking: {'time': 1, 'latitude': 721, 'longitude': 1440, 'level': 37}
• Chunk size (per variable): 154 MB

<details> <summary>Data summary table</summary>

name	short name	units	docs
100m_u_component_of_wind	u100	m s**-1	https://codes.ecmwf.int/grib/param-db/228246
100m_v_component_of_wind	v100	m s**-1	https://codes.ecmwf.int/grib/param-db/228247
10m_u_component_of_neutral_wind	u10n	m s**-1	https://codes.ecmwf.int/grib/param-db/228131
10m_u_component_of_wind	u10	m s**-1	https://codes.ecmwf.int/grib/param-db/165
10m_v_component_of_neutral_wind	v10n	m s**-1	https://codes.ecmwf.int/grib/param-db/228132
10m_v_component_of_wind	v10	m s**-1	https://codes.ecmwf.int/grib/param-db/166
10m_wind_gust_since_previous_post_processing	fg10	m s**-1	https://codes.ecmwf.int/grib/param-db/175049
2m_dewpoint_temperature	d2m	K	https://codes.ecmwf.int/grib/param-db/500018
2m_temperature	t2m	K	https://codes.ecmwf.int/grib/param-db/500013
air_density_over_the_oceans	p140209	kg m**-3	https://codes.ecmwf.int/grib/param-db/140209
angle_of_sub_gridscale_orography	anor	radians	https://codes.ecmwf.int/grib/param-db/162
anisotropy_of_sub_gridscale_orography	isor	~	https://codes.ecmwf.int/grib/param-db/161
benjamin_feir_index	bfi	dimensionless	https://codes.ecmwf.int/grib/param-db/140253
boundary_layer_dissipation	bld	J m**-2	https://codes.ecmwf.int/grib/param-db/145
boundary_layer_height	blh	m	https://codes.ecmwf.int/grib/param-db/159
charnock	chnk	~	https://codes.ecmwf.int/grib/param-db/148
clear_sky_direct_solar_radiation_at_surface	cdir	J m**-2	https://codes.ecmwf.int/grib/param-db/228022
cloud_base_height	cbh	m	https://codes.ecmwf.int/grib/param-db/228023
coefficient_of_drag_with_waves	cdww	dimensionless	https://codes.ecmwf.int/grib/param-db/140233
convective_available_potential_energy	cape	J kg**-1	https://codes.ecmwf.int/grib/param-db/59
convective_inhibition	cin	J kg**-1	https://codes.ecmwf.int/grib/param-db/228001
convective_precipitation	cp	m	https://codes.ecmwf.int/grib/param-db/228143
convective_rain_rate	crr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/228218
convective_snowfall	csf	m of water equivalent	https://codes.ecmwf.int/grib/param-db/239
convective_snowfall_rate_water_equivalent	csfr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/228220
downward_uv_radiation_at_the_surface	uvb	J m**-2	https://codes.ecmwf.int/grib/param-db/57
duct_base_height	dctb	m	https://codes.ecmwf.int/grib/param-db/228017
eastward_gravity_wave_surface_stress	lgws	N m**-2 s	https://codes.ecmwf.int/grib/param-db/195
eastward_turbulent_surface_stress	ewss	N m**-2 s	https://codes.ecmwf.int/grib/param-db/180
evaporation	e	m of water equivalent	https://codes.ecmwf.int/grib/param-db/182
forecast_albedo	fal	(0 - 1)	https://codes.ecmwf.int/grib/param-db/243
forecast_logarithm_of_surface_roughness_for_heat	flsr	~	https://codes.ecmwf.int/grib/param-db/245
forecast_surface_roughness	fsr	m	https://codes.ecmwf.int/grib/param-db/244
fraction_of_cloud_cover	cc	(0 - 1)	https://codes.ecmwf.int/grib/param-db/248
free_convective_velocity_over_the_oceans	p140208	m s**-1
friction_velocity	zust	m s**-1	https://codes.ecmwf.int/grib/param-db/228003
geopotential_at_surface	z	m2 s-2	https://codes.ecmwf.int/grib/param-db/129
gravity_wave_dissipation	gwd	J m**-2	https://codes.ecmwf.int/grib/param-db/197
high_cloud_cover	hcc	(0 - 1)	https://codes.ecmwf.int/grib/param-db/3075
high_vegetation_cover	cvh	(0 - 1)	https://codes.ecmwf.int/grib/param-db/28
ice_temperature_layer_1	istl1	K	https://codes.ecmwf.int/grib/param-db/35
ice_temperature_layer_2	istl2	K	https://codes.ecmwf.int/grib/param-db/36
ice_temperature_layer_3	istl3	K	https://codes.ecmwf.int/grib/param-db/37
ice_temperature_layer_4	istl4	K	https://codes.ecmwf.int/grib/param-db/38
instantaneous_10m_wind_gust	i10fg	m s**-1	https://codes.ecmwf.int/grib/param-db/228029
instantaneous_eastward_turbulent_surface_stress	iews	N m**-2	https://codes.ecmwf.int/grib/param-db/229
instantaneous_large_scale_surface_precipitation_fraction	ilspf	(0 - 1)	https://codes.ecmwf.int/grib/param-db/228217
instantaneous_moisture_flux	ie	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/232
instantaneous_northward_turbulent_surface_stress	inss	N m**-2	https://codes.ecmwf.int/grib/param-db/230
instantaneous_surface_sensible_heat_flux	ishf	W m**-2	https://codes.ecmwf.int/grib/param-db/231
k_index	kx	K	https://codes.ecmwf.int/grib/param-db/260121
lake_bottom_temperature	lblt	K	https://codes.ecmwf.int/grib/param-db/228010
lake_cover	cl	(0 - 1)	https://codes.ecmwf.int/grib/param-db/26
lake_depth	dl	m	https://codes.ecmwf.int/grib/param-db/228007
lake_ice_depth	licd	m	https://codes.ecmwf.int/grib/param-db/228014
lake_ice_temperature	lict	K	https://codes.ecmwf.int/grib/param-db/228013
lake_mix_layer_depth	lmld	m	https://codes.ecmwf.int/grib/param-db/228009
lake_mix_layer_temperature	lmlt	K	https://codes.ecmwf.int/grib/param-db/228008
lake_shape_factor	lshf	dimensionless	https://codes.ecmwf.int/grib/param-db/228012
lake_total_layer_temperature	ltlt	K	https://codes.ecmwf.int/grib/param-db/228011
land_sea_mask	lsm	(0 - 1)	https://codes.ecmwf.int/grib/param-db/172
large_scale_precipitation	lsp	m	https://codes.ecmwf.int/grib/param-db/3062
large_scale_precipitation_fraction	lspf	s	https://codes.ecmwf.int/grib/param-db/50
large_scale_rain_rate	lsrr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/228219
large_scale_snowfall	lsf	m of water equivalent	https://codes.ecmwf.int/grib/param-db/240
large_scale_snowfall_rate_water_equivalent	lssfr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/228221
leaf_area_index_high_vegetation	lai_hv	m2 m-2	https://codes.ecmwf.int/grib/param-db/67
leaf_area_index_low_vegetation	lai_lv	m2 m-2	https://codes.ecmwf.int/grib/param-db/66
low_cloud_cover	lcc	(0 - 1)	https://codes.ecmwf.int/grib/param-db/3073
low_vegetation_cover	cvl	(0 - 1)	https://codes.ecmwf.int/grib/param-db/27
maximum_2m_temperature_since_previous_post_processing	mx2t	K	https://codes.ecmwf.int/grib/param-db/201
maximum_individual_wave_height	hmax	m	https://codes.ecmwf.int/grib/param-db/140218
maximum_total_precipitation_rate_since_previous_post_processing	mxtpr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/228226
mean_boundary_layer_dissipation	mbld	W m**-2	https://codes.ecmwf.int/grib/param-db/235032
mean_convective_precipitation_rate	mcpr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/235030
mean_convective_snowfall_rate	mcsr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/235056
mean_direction_of_total_swell	mdts	degrees	https://codes.ecmwf.int/grib/param-db/140238
mean_direction_of_wind_waves	mdww	degrees	https://codes.ecmwf.int/grib/param-db/500072
mean_eastward_gravity_wave_surface_stress	megwss	N m**-2	https://codes.ecmwf.int/grib/param-db/235045
mean_eastward_turbulent_surface_stress	metss	N m**-2	https://codes.ecmwf.int/grib/param-db/235041
mean_evaporation_rate	mer	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/235043
mean_gravity_wave_dissipation	mgwd	W m**-2	https://codes.ecmwf.int/grib/param-db/235047
mean_large_scale_precipitation_fraction	mlspf	Proportion	https://codes.ecmwf.int/grib/param-db/235026
mean_large_scale_precipitation_rate	mlspr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/235029
mean_large_scale_snowfall_rate	mlssr	kg m**-2 s**-1	https://codes.ecmwf.int/grib/param-db/235057
mean_northward_gravity_wave_surface_stress	mngwss	N m**-2	https://codes.ecmwf.int/grib/param-db/235046
mean_northward_turbulent_surface_stress	mntss	N m**-2	https://codes.ecmwf.int/grib/param-db/235042
mean_period_of_total_swell	mpts	s	https://codes.ecmwf.int/grib/param-db/140239
mean_period_of_wind_waves	mpww	s	https://codes.ecmwf.int/grib/param-db/500074
mean_potential_evaporation_rate	mper	kg m**-2