Here, we'll use intake_zenodo_fetcher to populate the location an intake catalog points to with data from Zenodo.
For this to work, we need the catalog entries to contain a metadata.zenod_doi key.
This can increase reproducibility of workflows that use data archived on Zenodo.
An example for this FESOM2 example grid (https://zenodo.org/record/3865567) is:
metadata:
version: 1
plugins:
source:
- module: intake_xarray
sources:
FESOM2_sample:
driver: netcdf
description: 'FESOM2 pi mesh Sample dataset'
metadata:
zenodo_doi: "10.5281/zenodo.3865567"
args:
urlpath: "{{ CATALOG_DIR }}/FESOM2_PI_MESH/*.fesom.1948.nc"
xarray_kwargs:
decode_cf: False
combine: 'by_coords'
Then,
intake_zenodo_fetcher.download_zenodo_files_for_entry(
cat['FESOM2_sample']
)
will make sure all files matching the glob pattern "*.fesom.1948.nc" are downloaded to ./FESOM2_PI_MESH/ in the directory that also holds the catalog.
import intake
from pathlib import Path
from intake_zenodo_fetcher import download_zenodo_files_for_entry
# parameters
catalog_file = Path("catalogs/FESOM2_PI_MESH.yaml")
catalog = intake.open_catalog(str(catalog_file))
list(catalog)
['FESOM2_sample', 'MESH_NOD2D', 'MESH_ELEM2D', 'MESH_AUX3D']
display(catalog["FESOM2_sample"])
FESOM2_sample:
args:
urlpath: /home/jovyan/examples/catalogs//FESOM2_PI_MESH/*.fesom.1948.nc
xarray_kwargs:
combine: by_coords
decode_cf: false
description: FESOM2 pi mesh Sample dataset
driver: intake_xarray.netcdf.NetCDFSource
metadata:
catalog_dir: /home/jovyan/examples/catalogs/
zenodo_doi: 10.5281/zenodo.3865567
for catalog_entry in list(catalog):
download_zenodo_files_for_entry(
catalog[catalog_entry],
force_download=True
)
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/a_ice.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/a_ice.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/Av.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/Av.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/Kv.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/Kv.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/m_ice.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/m_ice.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/MLD1.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/MLD1.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/salt.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/salt.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/ssh.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/ssh.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/sst.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/sst.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/temp.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/temp.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/u.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/u.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/uice.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/uice.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/unod.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/unod.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/v.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/v.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/vice.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/vice.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/vnod.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/vnod.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/w.fesom.1948.nc to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/w.fesom.1948.nc
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/pi.zip to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/pi.zip
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/pi.zip to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/pi.zip
0%| | 0.00/100 [00:00<?, ?B/s]
will download https://zenodo.org/api/files/21cd530d-24d2-46bb-94c5-dc0ba924b5a6/pi.zip to /home/jovyan/examples/catalogs/FESOM2_PI_MESH/pi.zip
0%| | 0.00/100 [00:00<?, ?B/s]
Now we have fetched a local copy of all files from the Zenodo record https://zenodo.org/record/3865567 that are relevant for our catalog.
Let's make sure they are actually usable by reading them.
print(catalog["FESOM2_sample"].read())
<xarray.Dataset>
Dimensions: (elem: 5839, nod2: 3140, nz: 48, nz1: 47, time: 12)
Coordinates:
* time (time) float64 2.678e+06 5.097e+06 ... 2.886e+07 3.154e+07
Dimensions without coordinates: elem, nod2, nz, nz1
Data variables:
Av (time, elem, nz) float32 0.005085 0.0994 0.09765 ... 0.0 0.0 0.0
Kv (time, nod2, nz) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
MLD1 (time, nod2) float32 -75.2 -77.15 -80.05 ... -32.84 -33.67 -32.84
a_ice (time, nod2) float32 0.9829 0.9805 0.9803 0.9887 ... 0.0 0.0 0.0
m_ice (time, nod2) float32 1.13 1.111 1.107 1.163 ... 0.0 0.0 0.0 0.0
salt (time, nod2, nz1) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
ssh (time, nod2) float32 -0.5814 -0.5792 -0.5769 ... -1.576 -1.587
sst (time, nod2) float32 nan nan nan nan ... -1.517 -1.598 -1.536
temp (time, nod2, nz1) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
u (time, elem, nz1) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
uice (time, nod2) float32 -0.004919 -0.002244 0.0007494 ... 0.0 0.0 0.0
unod (time, nod2, nz1) float64 0.0008557 0.001066 0.001318 ... 0.0 0.0
v (time, elem, nz1) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
vice (time, nod2) float32 -0.00168 -0.001198 0.0005354 ... 0.0 0.0 0.0
vnod (time, nod2, nz1) float64 0.0006255 0.0005963 0.0005808 ... 0.0 0.0
w (time, nod2, nz) float32 nan nan nan nan nan ... 0.0 0.0 0.0 0.0
print(catalog["MESH_AUX3D"].read())
topo 0 -672 1 -534 2 -621 3 -744 4 -594 ... ... 3135 -1200 3136 -850 3137 -560 3138 -121 3139 -209 [3140 rows x 1 columns]
print(catalog["MESH_NOD2D"].read())
node_number x y flag 0 1 267.4665 84.5252 0 1 2 270.9702 84.3700 0 2 3 268.3207 84.0503 0 3 4 263.8669 84.5900 0 4 5 264.3323 84.2140 0 ... ... ... ... ... 3135 3136 139.7089 -79.0173 0 3136 3137 144.1223 -79.2533 0 3137 3138 148.0000 -80.3240 1 3138 3139 135.7037 -79.9329 1 3139 3140 141.5839 -80.1193 1 [3140 rows x 4 columns]
print(catalog["MESH_ELEM2D"].read())
first_elem second_elem third_elem 0 1 12 2 1 2 12 10 2 2 10 9 3 3 1 2 4 3 5 1 ... ... ... ... 5834 3138 3137 3132 5835 3138 3131 3133 5836 3139 3136 3140 5837 3139 3135 3134 5838 3140 3137 3138 [5839 rows x 3 columns]