After this introduction, you will know how to:
GriddedData
and UngriddedData
)ColocatedData
)pandas
or xarray
import pyaerocom as pya
import matplotlib.pyplot as plt
from warnings import filterwarnings
filterwarnings('ignore')
pya.change_verbosity('critical', log=pya.const.print_log) # don't output warnings
pya.__version__
'0.8.1.dev3'
Please make sure to use version 0.8.1.dev2 (or later). If you have an earlier version, follow these instructions in order to update your installation.
import socket
if socket.gethostname() == 'pc4971':
print('I am on Jonas PC')
DATA_BASEDIR = '/home/jonasg/MyPyaerocom/pyaerocom-testdata'
else:
print('I assume I am on the Abisko Jupyter hub')
DATA_BASEDIR = '/home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/'
I assume I am on the Abisko Jupyter hub
pya.const.BASEDIR = DATA_BASEDIR
pya.const.BASEDIR
'/home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/'
browse_database
)¶pya.browse_database('*TM5*')
Pyaerocom ReadGridded --------------------- Data ID: TM5_AP3-CTRL2019 Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/TM5_AP3-CTRL2019/renamed Available experiments: ['AP3-CTRL2019'] Available years: [2010] Available frequencies ['monthly'] Available variables: ['abs350aer', 'abs440aer', 'abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs870aer', 'abs870dryaer', 'airmass', 'asyaer', 'asydryaer', 'depbc', 'depdms', 'depdust', 'dephno3', 'depmsa', 'depn', 'depnh3', 'depnh4', 'depnhx', 'depno2', 'depno3', 'depnoy', 'depo3', 'depoa', 'deps', 'depso2', 'depso4', 'depss', 'dh', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'dryno3', 'drynoy', 'dryo3', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550drylt1aer', 'ec870dryaer', 'emibc', 'emico', 'emidms', 'emidust', 'emiisop', 'emin', 'eminh3', 'eminox', 'emioa', 'emis', 'emiso2', 'emiso4', 'emiss', 'emiterp', 'emivoc', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od350aer', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550lt1ss', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pr', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcnh4', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'ta', 'temp', 'vmrch4', 'vmrco', 'vmrno', 'vmrno2', 'vmro3', 'vmroh', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetmsa', 'wetnh3', 'wetnh4', 'wetno3', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss']
You can use the *Data ID* or the *Data directory* to read this dataset (next step)
ReadGridded
class)¶DATA_ID = 'TM5_AP3-CTRL2019'
reader = pya.io.ReadGridded(DATA_ID)
print(reader)
Pyaerocom ReadGridded --------------------- Data ID: TM5_AP3-CTRL2019 Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/TM5_AP3-CTRL2019/renamed Available experiments: ['AP3-CTRL2019'] Available years: [2010] Available frequencies ['monthly'] Available variables: ['abs350aer', 'abs440aer', 'abs440dryaer', 'abs550aer', 'abs550dryaer', 'abs550drylt1aer', 'abs870aer', 'abs870dryaer', 'airmass', 'asyaer', 'asydryaer', 'depbc', 'depdms', 'depdust', 'dephno3', 'depmsa', 'depn', 'depnh3', 'depnh4', 'depnhx', 'depno2', 'depno3', 'depnoy', 'depo3', 'depoa', 'deps', 'depso2', 'depso4', 'depss', 'dh', 'drybc', 'drydms', 'drydust', 'dryhno3', 'drynh3', 'dryno2', 'dryno3', 'drynoy', 'dryo3', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550drylt1aer', 'ec870dryaer', 'emibc', 'emico', 'emidms', 'emidust', 'emiisop', 'emin', 'eminh3', 'eminox', 'emioa', 'emis', 'emiso2', 'emiso4', 'emiss', 'emiterp', 'emivoc', 'hus', 'loadbc', 'loaddust', 'loadno3', 'loadoa', 'loadso4', 'loadss', 'od350aer', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1dust', 'od550lt1ss', 'od550no3', 'od550oa', 'od550so4', 'od550ss', 'od870aer', 'pr', 'sconcbc', 'sconcdust', 'sconcmsa', 'sconcnh4', 'sconcno3', 'sconcoa', 'sconcso4', 'sconcss', 'ta', 'temp', 'vmrch4', 'vmrco', 'vmrno', 'vmrno2', 'vmro3', 'vmroh', 'wetbc', 'wetdms', 'wetdust', 'wethno3', 'wetmsa', 'wetnh3', 'wetnh4', 'wetno3', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss']
pya.get_variable('od550aer')
od550aer standard_name: atmosphere_optical_thickness_due_to_ambient_aerosol_particles; Unit: 1
help(reader.read_var)
Help on method read_var in module pyaerocom.io.readgridded: read_var(var_name, start=None, stop=None, ts_type=None, experiment=None, vert_which=None, flex_ts_type=True, prefer_longer=False, aux_vars=None, aux_fun=None, **kwargs) method of pyaerocom.io.readgridded.ReadGridded instance Read model data for a specific variable This method searches all valid files for a given variable and for a provided temporal resolution (e.g. *daily, monthly*), optionally within a certain time window, that may be specified on class instantiation or using the corresponding input parameters provided in this method. The individual NetCDF files for a given temporal period are loaded as instances of the :class:`iris.Cube` object and appended to an instance of the :class:`iris.cube.CubeList` object. The latter is then used to concatenate the individual cubes in time into a single instance of the :class:`pyaerocom.GriddedData` class. In order to ensure that this works, several things need to be ensured, which are listed in the following and which may be controlled within the global settings for NetCDF import using the attribute :attr:`GRID_IO` (instance of :class:`OnLoad`) in the default instance of the :class:`pyaerocom.config.Config` object accessible via ``pyaerocom.const``. Parameters ---------- var_name : str variable that are supposed to be read start : Timestamp or str, optional start time of data import stop : Timestamp or str, optional stop time of data import ts_type : str string specifying temporal resolution (choose from "hourly", "3hourly", "daily", "monthly"). If None, prioritised of the available resolutions is used experiment : str name of experiment (only relevant if this dataset contains more than one experiment) vert_which : str or dict, optional valid AeroCom vertical info string encoded in name (e.g. Column, ModelLevel) or dictionary containing var_name as key and vertical coded string as value, accordingly flex_ts_type : bool if True and if applicable, then another ts_type is used in case the input ts_type is not available for this variable prefer_longer : bool if True and applicable, the ts_type resulting in the longer time coverage will be preferred over other possible frequencies that match the query. aux_vars : list only relevant if `var_name` is not available for reading but needs to be computed: list of variables that are required to compute `var_name` aux_fun : callable only relevant if `var_name` is not available for reading but needs to be computed: custom method for computation (cf. :func:`add_aux_compute` for details) **kwargs additional keyword args parsed to :func:`_load_var` Returns ------- GriddedData loaded data object Raises ------ AttributeError if none of the ts_types identified from file names is valid VarNotAvailableError if specified ts_type is not supported
model_data = reader.read_var('od550aer', start=2010)
type(model_data)
pyaerocom.griddeddata.GriddedData
GriddedData
object¶model_data.ts_type # temporal resolution
'monthly'
model_data.quickplot_map(11);
GriddedData
to iris.Cube
or xarray.DataArray
¶Actually, the GriddedData
object is based on the iris.Cube
object which can be accessed via the cube
attr:
cube = model_data.cube
type(cube)
iris.cube.Cube
cube
Atmosphere Optical Thickness Due To Ambient Aerosol (1) | time | latitude | longitude |
---|---|---|---|
Shape | 12 | 90 | 120 |
Dimension coordinates | |||
time | x | - | - |
latitude | - | x | - |
longitude | - | - | x |
Attributes | |||
Conventions | CF-1.6 | ||
computed | False | ||
concatenated | False | ||
contact | Twan van Noije (noije@knmi.nl) | ||
data_id | TM5_AP3-CTRL2019 | ||
experiment_id | AP3-CTRL2019 | ||
from_files | ['/home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/TM... | ||
institute_id | KNMI | ||
institution | Royal Netherlands Meteorological Institute, De Bilt, The Netherlands | ||
model_id | TM5 | ||
outliers_removed | False | ||
project_id | AeroCom Phase 3 | ||
reader | None | ||
references | Van Noije, T.P.C., et al. (Geosci. Model Dev., 7, 2435-2475, 2014); Bergman,... | ||
region | None | ||
regridded | False | ||
source | TM5-mp, r1058: CTM ERA-Interim 3x2 34L | ||
title | TM5 model output prepared for AeroCom Phase 3 | ||
ts_type | monthly | ||
var_name_read | n/d | ||
Cell methods | |||
point | longitude, latitude | ||
mean | time |
try: # pyaerocom >= 0.8.1
data_arr = model_data.to_xarray()
except: # pyaerocom < 0.8.1
print('Coming soon in pyaerocom v0.8.1')
import xarray as xarr
data_arr = xarr.DataArray.from_iris(model_data.cube)
data_arr
<xarray.DataArray 'od550aer' (time: 12, lat: 90, lon: 120)> dask.array<filled, shape=(12, 90, 120), dtype=float32, chunksize=(12, 90, 120), chunktype=numpy.ndarray> Coordinates: * time (time) object 2010-01-15 12:00:00 ... 2010-12-15 12:00:00 * lat (lat) float64 -89.0 -87.0 -85.0 -83.0 -81.0 ... 83.0 85.0 87.0 89.0 * lon (lon) float64 1.5 4.5 7.5 10.5 13.5 ... 349.5 352.5 355.5 358.5 Attributes: standard_name: atmosphere_optical_thickness_due_to_ambient_aerosol long_name: Ambient Aerosol Optical Thickness at 550 nm institution: Royal Netherlands Meteorological Institute, De Bilt, T... institute_id: KNMI source: TM5-mp, r1058: CTM ERA-Interim 3x2 34L model_id: TM5 references: Van Noije, T.P.C., et al. (Geosci. Model Dev., 7, 2435... experiment_id: AP3-CTRL2019 project_id: AeroCom Phase 3 title: TM5 model output prepared for AeroCom Phase 3 Conventions: CF-1.6 contact: Twan van Noije (noije@knmi.nl) from_files: ['/home/notebook/shared-ns1000k/inputs/pyaerocom-testd... data_id: TM5_AP3-CTRL2019 var_name_read: n/d ts_type: monthly regridded: False outliers_removed: False computed: False concatenated: False cell_methods: longitude: latitude: point time: mean
GriddedData
object that may be useful¶LON_ABISKO = 18.8312 #° E
LAT_ABISKO = 68.3495 #° N
subset_abisko = model_data.sel(longitude=LON_ABISKO, latitude=LAT_ABISKO)
subset_abisko
pyaerocom.GriddedData Grid data: <iris 'Cube' of atmosphere_optical_thickness_due_to_ambient_aerosol / (1) (time: 12)>
try: # pyaerocom >= 0.8.1
subset_abisko.to_xarray().plot();
except: # pyaerocom < 0.8.1
print('Coming soon in pyaerocom v0.8.1')
import xarray as xarr
data_arr = xarr.DataArray.from_iris(subset_abisko.cube).plot()
ReadUngridded
class)¶pya.browse_database('Aeronet*V3*Lev2*')
Dataset name: AeronetSunV3Lev2.daily Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata//obsdata/AeronetSunV3Lev2.0.daily/renamed Supported variables: ['od340aer', 'od440aer', 'od500aer', 'od870aer', 'ang4487aer', 'ang4487aer_calc', 'od550aer'] Last revision: 20190920 Dataset name: AeronetSDAV3Lev2.daily Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata//obsdata/Aeronet.SDA.V3L2.0.daily/renamed Supported variables: ['od500gt1aer', 'od500lt1aer', 'od500aer', 'ang4487aer', 'od550aer', 'od550gt1aer', 'od550lt1aer'] Last revision: 20190920 Reading failed for AeronetSDAV3Lev2.AP. Error: NetworkNotImplemented('No reading class available yet for dataset AeronetSDAV3Lev2.AP')
OBS_ID = 'AeronetSunV3Lev2.daily'
obs_reader = pya.io.ReadUngridded(OBS_ID)
print(obs_reader)
Dataset name: AeronetSunV3Lev2.daily Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata//obsdata/AeronetSunV3Lev2.0.daily/renamed Supported variables: ['od340aer', 'od440aer', 'od500aer', 'od870aer', 'ang4487aer', 'ang4487aer_calc', 'od550aer'] Last revision: 20190920
obs_data = obs_reader.read(vars_to_retrieve=['od550aer'])
obs_data
UngriddedData <networks: ['AeronetSunV3Lev2.daily']; vars: ['od550aer']; instruments: ['sun_photometer'];No. of stations: 1230
UngriddedData
object¶ax = obs_data.plot_station_coordinates(markersize=16)
ax = obs_data.plot_station_coordinates(var_name='od550aer',
start=2008, stop=2012,
markersize=12, color='lime', ax=ax)
len(obs_data.metadata)
1230
e.g. metadata of first file:
obs_data.metadata[420]
OrderedDict([('var_info', OrderedDict([('od550aer', OrderedDict([('units', '1')]))])), ('latitude', 19.932449999999996), ('longitude', 99.04539999999999), ('altitude', 1536.0), ('station_name', 'Doi_Ang_Khang'), ('PI', 'Brent_Holben'), ('ts_type', 'daily'), ('data_id', 'AeronetSunV3Lev2.daily'), ('variables', ['od550aer']), ('instrument_name', 'sun_photometer'), ('data_revision', '20190920')])
obs_arctic = obs_data.apply_filters(latitude=[70, 90])
obs_arctic.plot_station_coordinates(markersize=40, marker='x', color='r');
lille_data = obs_data.to_station_data('*Lille*')
type(lille_data)
pyaerocom.stationdata.StationData
StationData
¶.
and []
to access and assign attributes)pandas.Series
lille_data.keys()
odict_keys(['dtime', 'var_info', 'station_coords', 'data_err', 'overlap', 'data_flagged', 'filename', 'station_id', 'station_name', 'instrument_name', 'PI', 'country', 'ts_type', 'latitude', 'longitude', 'altitude', 'data_id', 'dataset_name', 'data_product', 'data_version', 'data_level', 'revision_date', 'website', 'ts_type_src', 'stat_merge_pref_attr', 'data_revision', 'od550aer'])
lille_data.longitude
3.1416669999999987
lille_data['latitude']
50.611667000000004
pandas.Series
simply by using the variable name¶lille_data.od550aer # THIS is exactly equivalent to the command lille_data['od550aer']
1995-06-30 12:00:00 0.403992 1995-07-01 12:00:00 0.695444 1995-07-04 12:00:00 0.213412 1995-07-07 12:00:00 0.757272 1995-07-08 12:00:00 1.072732 ... 2019-06-16 12:00:00 0.061789 2019-06-17 12:00:00 0.100830 2019-06-18 12:00:00 0.130222 2019-06-20 12:00:00 0.046714 2019-06-21 12:00:00 0.074854 Length: 3399, dtype: float64
ax = lille_data.plot_timeseries(var_name='od550aer', marker='x', linestyle='none')
ax = lille_data.plot_timeseries(var_name='od550aer', ts_type='monthly', linestyle='-', lw=3, ax=ax)
col_data = pya.colocation.colocate_gridded_ungridded(model_data,
obs_data,
ts_type='monthly',
start=2010)
help(pya.colocation.colocate_gridded_gridded)
Help on function colocate_gridded_gridded in module pyaerocom.colocation: colocate_gridded_gridded(gridded_data, gridded_data_ref, ts_type=None, start=None, stop=None, filter_name=None, regrid_res_deg=None, remove_outliers=True, vert_scheme=None, harmonise_units=True, regrid_scheme='areaweighted', var_outlier_ranges=None, var_ref_outlier_ranges=None, update_baseyear_gridded=None, apply_time_resampling_constraints=None, min_num_obs=None, colocate_time=False, var_keep_outliers=True, var_ref_keep_outliers=False, **kwargs) Colocate 2 gridded data objects Todo ---- - think about vertical dimension (vert_scheme input not used at the moment) Parameters ---------- gridded_data : GriddedData gridded data (e.g. model results) gridded_data_ref : GriddedData reference dataset that is used to evaluate :attr:`gridded_data` (e.g. gridded observation data) ts_type : str desired temporal resolution of colocated data (must be valid AeroCom ts_type str such as daily, monthly, yearly..) start : :obj:`str` or :obj:`datetime64` or similar, optional start time for colocation, if None, the start time of the input :class:`GriddedData` object is used stop : :obj:`str` or :obj:`datetime64` or similar, optional stop time for colocation, if None, the stop time of the input :class:`GriddedData` object is used filter_name : str string specifying filter used (cf. :class:`pyaerocom.filter.Filter` for details). If None, then it is set to 'WORLD-wMOUNTAINS', which corresponds to no filtering (world with mountains). Use WORLD-noMOUNTAINS to exclude mountain sites. regrid_res_deg : :obj:`int`, optional regrid resolution in degrees. If specified, the input gridded data objects will be regridded in lon / lat dimension to the input resolution. (BETA feature) remove_outliers : bool if True, outliers are removed from model and obs data before colocation, else not. vert_scheme : str string specifying scheme used to reduce the dimensionality in case input grid data contains vertical dimension. Example schemes are `mean, surface, altitude`, for details see :func:`GriddedData.to_time_series`. harmonise_units : bool if True, units are attempted to be harmonised (note: raises Exception if True and units cannot be harmonised). regrid_scheme : str iris scheme used for regridding (defaults to area weighted regridding) var_outlier_ranges : :obj:`dict`, optional dictionary specifying outlier ranges for dataset to be analysed (e.g. dict(od550aer = [-0.05, 10], ang4487aer=[0,4])). If None, then the pyaerocom default outlier ranges are used for the input variable. Defaults to None. var_ref_outlier_ranges : dict, optional like `var_outlier_ranges` but for reference dataset. update_baseyear_gridded : int, optional optional input that can be set in order to redefine the time dimension in the gridded data object to be analysed. E.g., if the data object is a climatology (one year of data) that has set the base year of the time dimension to a value other than the specified input start / stop time this may be used to update the time in order to make colocation possible. apply_time_resampling_constraints : bool, optional if True, then time resampling constraints are applied as provided via :attr:`min_num_obs` or if that one is unspecified, as defined in :attr:`pyaerocom.const.OBS_MIN_NUM_RESAMPLE`. If None, than :attr:`pyaerocom.const.OBS_APPLY_TIME_RESAMPLE_CONSTRAINTS` is used (which defaults to True !!). min_num_obs : int or dict, optional minimum number of observations for resampling of time colocate_time : bool if True and if original time resolution of data is higher than desired time resolution (`ts_type`), then both datasets are colocated in time *before* resampling to lower resolution. var_keep_outliers : bool if True, then no outliers will be removed from dataset to be analysed, even if `remove_outliers` is True. That is because for model evaluation often only outliers are supposed to be removed in the observations but not in the model. var_ref_keep_outliers : bool if True, then no outliers will be removed from the reference dataset, even if `remove_outliers` is True. **kwargs additional keyword args (not used here, but included such that factory class can handle different methods with different inputs) Returns ------- ColocatedData instance of colocated data
type(col_data)
pyaerocom.colocateddata.ColocatedData
col_data.plot_scatter(marker='o', color='blue', alpha=0.1);
xarray.DataArray
object can be simply accessed via¶col_data.data
<xarray.DataArray 'od550aer' (data_source: 2, time: 12, station_name: 290)> array([[[ nan, 0.11758773, nan, ..., nan, nan, 0.22213848], [ nan, 0.13212825, nan, ..., 0.12098059, nan, 0.4297616 ], [ nan, 0.14685542, nan, ..., 0.12334401, nan, 0.91985307], ..., [0.11797653, 0.11601176, nan, ..., 0.05336442, 0.03394842, 0.37093354], [0.13223569, 0.19505731, nan, ..., nan, nan, 0.26176478], [ nan, nan, nan, ..., nan, nan, 0.37904951]], [[0.14947775, 0.14656237, 0.16371733, ..., 0.08020622, 0.03766838, 0.23015726], [0.14588432, 0.2863543 , 0.20080711, ..., 0.11745247, 0.06611011, 0.46402201], [0.19967574, 0.28439081, 0.91542172, ..., 0.13032669, 0.05874216, 1.46623838], ..., [0.21575473, 0.23416109, 0.34703311, ..., 0.09446716, 0.05008808, 0.51150841], [0.16433115, 0.19931237, 0.38158169, ..., 0.10818329, 0.03546635, 0.40573141], [0.10476176, 0.13849808, 0.30405122, ..., 0.14951633, 0.04037524, 0.45553884]]]) Coordinates: * data_source (data_source) <U22 'AeronetSunV3Lev2.daily' 'TM5_AP3-CTRL2019' var_name (data_source) <U8 'od550aer' 'od550aer' var_units (data_source) <U1 '1' '1' ts_type_src (data_source) <U7 'daily' 'monthly' * time (time) datetime64[ns] 2010-01-01 2010-02-01 ... 2010-12-01 * station_name (station_name) <U19 'ARM_Darwin' ... 'Zinder_Airport' latitude (station_name) float64 -12.43 37.97 15.35 ... 62.45 13.78 longitude (station_name) float64 130.9 23.72 -1.479 ... -114.4 8.99 altitude (station_name) float64 29.9 130.0 305.0 ... 300.0 220.8 456.0 Attributes: data_source: ['AeronetSunV3Lev2.daily', 'TM5_AP3-CTRL2019'] var_name: ['od550aer', 'od550aer'] ts_type: monthly filter_name: WORLD-wMOUNTAINS ts_type_src: ['daily', 'monthly'] start_str: 20100101 stop_str: 20101231 var_units: ['1', '1'] vert_scheme: None data_level: 3 revision_ref: 20190920 from_files: ['aerocom3_TM5_AP3-CTRL2019_od550aer_Column_2010_mont... from_files_ref: None stations_ignored: None colocate_time: False apply_constraints: True min_num_obs: {'yearly': {'monthly': 3}, 'monthly': {'daily': 7}, '... outliers_removed: True region: WORLD lon_range: [-180, 180] lat_range: [-90, 90] alt_range: None
ColocatedData
object to NetCDF¶col_data.savename_aerocom
'od550aer_REF-AeronetSunV3Lev2.daily_MOD-TM5_AP3-CTRL2019_20100101_20101231_monthly_WORLD-wMOUNTAINS'
col_data.to_netcdf('.')
The example above did essentially the following steps:
GriddedData
and UngriddedData
, respectivelyColocatedData
objectAll these steps can be done with a one-liner in pyaerocom using the Colocator
class
Colocate the same model and observation network but now use the Angstrom exponent (ang4487aer) instead of the AOD:
colocator = pya.Colocator(model_id=DATA_ID,
obs_id=OBS_ID,
obs_vars='ang4487aer',
ts_type='monthly',
start=2010,
basedir_coldata='.',
reanalyse_existing=True,
save_coldata=True)
colocator.run()
Reading files 1-124 of 1230 (ReadAeronetSunV3) | 00:05:30 (delta = 0 s') Reading file 0 of 1230 (ReadAeronetSunV3) Reading files 124-247 of 1230 (ReadAeronetSunV3) | 00:05:33 (delta = 3 s') Reading file 123 of 1230 (ReadAeronetSunV3) Reading files 247-370 of 1230 (ReadAeronetSunV3) | 00:05:36 (delta = 2 s') Reading file 246 of 1230 (ReadAeronetSunV3) Reading files 370-493 of 1230 (ReadAeronetSunV3) | 00:05:38 (delta = 2 s') Reading file 369 of 1230 (ReadAeronetSunV3) Reading files 493-616 of 1230 (ReadAeronetSunV3) | 00:05:42 (delta = 3 s') Reading file 492 of 1230 (ReadAeronetSunV3) Reading files 616-739 of 1230 (ReadAeronetSunV3) | 00:05:46 (delta = 4 s') Reading file 615 of 1230 (ReadAeronetSunV3) Reading files 739-862 of 1230 (ReadAeronetSunV3) | 00:05:51 (delta = 4 s') Reading file 738 of 1230 (ReadAeronetSunV3) Reading files 862-985 of 1230 (ReadAeronetSunV3) | 00:05:55 (delta = 4 s') Reading file 861 of 1230 (ReadAeronetSunV3) Reading files 985-1108 of 1230 (ReadAeronetSunV3) | 00:05:59 (delta = 4 s') Reading file 984 of 1230 (ReadAeronetSunV3) Reading files 1108-1231 of 1230 (ReadAeronetSunV3) | 00:06:05 (delta = 5 s') Reading file 1107 of 1230 (ReadAeronetSunV3)
The Colocator
objects stores all ColocatedData
objects that were created in it's data
attribute, which is a nested dictionary, organised via model_id
and var_name
:
colocator.data['TM5_AP3-CTRL2019']['ang4487aer']
<xarray.DataArray 'ang4487aer' (data_source: 2, time: 12, station_name: 264)> array([[[ nan, 1.128966 , nan, ..., nan, nan, 0.25999457], [ nan, 0.89081575, nan, ..., 1.12947156, nan, 0.30509289], [ nan, 1.12122509, nan, ..., 1.14267986, nan, 0.13727079], ..., [0.97128121, 1.2908889 , nan, ..., 1.56412958, 1.0926709 , 0.39523561], [1.34621412, 0.934353 , nan, ..., nan, nan, 0.49452866], [ nan, nan, nan, ..., nan, nan, 0.4892751 ]], [[0.86711586, 0.73773724, 0.52013141, ..., 1.39102662, 1.35547209, 0.62231678], [0.98289979, 0.30570188, 0.58608049, ..., 1.23340809, 1.23590493, 0.44316554], [0.57381302, 0.59244061, 0.15318914, ..., 1.31365597, 1.16984105, 0.19862083], ..., [0.8841635 , 0.85821074, 1.01289105, ..., 1.36450207, 1.38448656, 0.95568919], [1.28473127, 0.65777272, 0.54912972, ..., 1.26303518, 1.3864181 , 0.62829041], [1.14279449, 0.75016189, 0.47216135, ..., 1.05199301, 1.37411582, 0.41307265]]]) Coordinates: * data_source (data_source) <U22 'AeronetSunV3Lev2.daily' 'TM5_AP3-CTRL2019' var_name (data_source) <U10 'ang4487aer' 'ang4487aer' var_units (data_source) <U1 '1' '1' ts_type_src (data_source) <U7 'daily' 'monthly' * time (time) datetime64[ns] 2010-01-01 2010-02-01 ... 2010-12-01 * station_name (station_name) <U19 'ARM_Darwin' ... 'Zinder_Airport' latitude (station_name) float64 -12.43 37.97 15.35 ... 62.45 13.78 longitude (station_name) float64 130.9 23.72 -1.479 ... -114.4 8.99 altitude (station_name) float64 29.9 130.0 305.0 ... 300.0 220.8 456.0 Attributes: data_source: ['AeronetSunV3Lev2.daily', 'TM5_AP3-CTRL2019'] var_name: ['ang4487aer', 'ang4487aer'] ts_type: monthly filter_name: WORLD-noMOUNTAINS ts_type_src: ['daily', 'monthly'] start_str: 20100101 stop_str: 20101231 var_units: ['1', '1'] vert_scheme: None data_level: 3 revision_ref: 20190920 from_files: ['aerocom3_TM5_AP3-CTRL2019_od440aer_Column_2010_mont... from_files_ref: None stations_ignored: None colocate_time: 0 apply_constraints: 1 outliers_removed: 1 region: WORLD lon_range: [-180, 180] lat_range: [-90, 90] alt_range: [-1000000.0, 1000.0] _min_num_obs: yearly,monthly,3;monthly,daily,7;daily,hourly,6;hourl...
colocator.data[DATA_ID]['ang4487aer'].plot_scatter(marker='o', alpha=0.1, color='g');
pya.browse_database('*CAM*')
Pyaerocom ReadGridded --------------------- Data ID: ECMWF-IFS-CY45R1-CAMS-CTRL-met2010_AP3-CTRL Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/ECMWF-IFS-CY45R1-CAMS-CTRL-met2010_AP3-CTRL/renamed Available experiments: ['AP3-CTRL'] Available years: [1850, 2010] Available frequencies ['3hourly' 'daily' 'monthly'] Available variables: ['abs550aer', 'clh', 'cll', 'clm', 'clt', 'depbc', 'depdu', 'depoa', 'depso2', 'depso4', 'depss', 'drybc', 'drydu', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'emibc', 'emidu', 'emioa', 'emiso2', 'emiss', 'iwp', 'loadbc', 'loaddu', 'loadlt10du', 'loadlt10ss', 'loadlt1du', 'loadlt1ss', 'loadlt25du', 'loadlt25ss', 'loadoa', 'loadso2', 'loadso4', 'loadss', 'lwp', 'od440aer', 'od550aer', 'od550bc', 'od550du', 'od550lt10du', 'od550lt10ss', 'od550lt1du', 'od550lt1ss', 'od550lt25du', 'od550lt25ss', 'od550oa', 'od550so4', 'od550ss', 'od865aer', 'prodso4', 'ps', 'sconcbc', 'sconcdu', 'sconcoa', 'sconcpm1', 'sconcpm10', 'sconcpm25', 'sconcso2', 'sconcso4', 'sconcss', 'wetbc', 'wetdu', 'wetoa', 'wetso2', 'wetso4', 'wetss'] Pyaerocom ReadGridded --------------------- Data ID: CAM5-ATRAS_AP3-CTRL Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/CAM5-ATRAS_AP3-CTRL/renamed Available experiments: ['AP3-CTRL'] Available years: [1750, 1850, 2010] Available frequencies ['daily' 'monthly'] Available variables: ['abs440aer', 'abs440bc', 'abs440dryaer', 'abs440dust', 'abs440oc', 'abs550aer', 'abs550bc', 'abs550dryaer', 'abs550dust', 'abs550lt1aer', 'abs550oc', 'abs550rh40aer', 'abs550rh85aer', 'abs870aer', 'abs870bc', 'abs870dust', 'abs870oc', 'absc550aer', 'airmass', 'areacella', 'bldep', 'cdnc', 'cheaqpso4', 'chegpso4', 'chepsoa', 'cli', 'clivi', 'clt', 'clw', 'co', 'convclt', 'dh', 'dms', 'drybc', 'drydust', 'drynh3', 'drynh4', 'drynoy', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec1000dust', 'ec10umdust', 'ec440dryaer', 'ec550aer', 'ec550dryaer', 'ec550dust', 'ec550rh40aer', 'ec550rh85aer', 'emibc', 'emibvoc', 'emico', 'emidms', 'emidust', 'emiisop', 'emimtp', 'eminh3', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'emivoc', 'hcho', 'hfls', 'hfss', 'hno3', 'hus', 'isop', 'loadbc', 'loaddms', 'loaddu', 'loadhno3', 'loadlt10du', 'loadlt1du', 'loadlt1ss', 'loadlt25du', 'loadlt25ss', 'loadnh3', 'loadnh4', 'loadno3', 'loadoa', 'loadso2', 'loadso4', 'loadsoa', 'loadss', 'lwp', 'mmraerh2o', 'mmrbc', 'mmrdust', 'mmrnh4', 'mmrno3', 'mmroa', 'mmrpm1', 'mmrpm10', 'mmrpm2p5', 'mmrso4', 'mmrsoa', 'mmrss', 'mtp', 'nh3', 'od1000dust', 'od10umdust', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt10du', 'od550lt1aer', 'od550lt1du', 'od550lt1ss', 'od550lt2p5du', 'od550lt2p5ss', 'od550no3', 'od550oa', 'od550so4', 'od550soa', 'od550ss', 'od870aer', 'orog', 'pfull', 'phalf', 'pr', 'prc', 'ps', 'ptp', 'rh', 'rho', 'rlds', 'rldscs', 'rlus', 'rluscs', 'rlut', 'rlutca', 'rlutcs', 'rlutcsca', 'rsds', 'rsdscs', 'rsdt', 'rsus', 'rsut', 'rsutca', 'rsutcs', 'rsutcsca', 'sftlf', 'so2', 'ta', 'tas', 'tos', 'ts', 'ua', 'va', 'wa', 'wetbc', 'wetdust', 'wetnh3', 'wetnh4', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss', 'ztp'] Pyaerocom ReadGridded --------------------- Data ID: ECMWF_CAMS_REAN Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/ECMWF_CAMS_REAN/renamed Available experiments: ['REAN'] Available years: [2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018] Available frequencies ['daily'] Available variables: ['ang4487aer', 'od550aer', 'od550oa', 'od550so4'] Pyaerocom ReadGridded --------------------- Data ID: CAM5-ATRAS_AP3-HIST Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/CAM5-ATRAS_AP3-HIST/renamed Available experiments: ['AP3-HIST'] Available years: [1750, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920, 1930, 1940, 1950, 1960, 1970, 1980, 1985, 1990, 1995, 2000, 2005, 2010, 2015] Available frequencies ['monthly' 'daily'] Available variables: ['abs550aer', 'abs550dryaer', 'airmass', 'chepsoa', 'clt', 'dh', 'drybc', 'drydust', 'drynh3', 'drynh4', 'drynoy', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'eminh3', 'eminox', 'emioa', 'emiso2', 'emiso4', 'emiss', 'loadbc', 'loaddms', 'loaddu', 'loadhno3', 'loadlt1du', 'loadlt1ss', 'loadlt25du', 'loadlt25ss', 'loadnh3', 'loadnh4', 'loadno3', 'loadoa', 'loadso2', 'loadso4', 'loadsoa', 'loadss', 'mmraerh2o', 'mmrbc', 'mmrdust', 'mmrnh4', 'mmrno3', 'mmroa', 'mmrpm1', 'mmrpm10', 'mmrpm2p5', 'mmrso4', 'mmrsoa', 'mmrss', 'od440aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550dust', 'od550lt1aer', 'od550lt1du', 'od550lt1ss', 'od550lt2p5du', 'od550lt2p5ss', 'od550no3', 'od550oa', 'od550so4', 'od550soa', 'od550ss', 'od870aer', 'pfull', 'rh', 'rsds', 'rsus', 'rsut', 'rsutca', 'rsutcs', 'rsutcsca', 'so2', 'wetbc', 'wetdust', 'wetnh3', 'wetnh4', 'wetnoy', 'wetoa', 'wetso2', 'wetso4', 'wetss'] Pyaerocom ReadGridded --------------------- Data ID: CAM53-Oslo_7310_MG15CLM45_5feb2017IHK_53OSLO_PD_UNTUNED Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/CAM53-Oslo_7310_MG15CLM45_5feb2017IHK_53OSLO_PD_UNTUNED/renamed Available experiments: ['UNTUNED'] Available years: [2008, 2009, 2010] Available frequencies ['monthly'] Available variables: ['abs550aer', 'abs550aercs', 'ang4487aer', 'ang4487csaer', 'od550aer', 'od550csaer', 'od550lt1aer'] Pyaerocom ReadGridded --------------------- Data ID: CAM6-Oslo_NHIST_f19_tn14_20190710_2010 Data directory: /home/notebook/shared-ns1000k/inputs/pyaerocom-testdata/modeldata/CAM6-Oslo_NHIST_f19_tn14_20190710_2010/renamed Available experiments: ['20190710'] Available years: [2010] Available frequencies ['monthly'] Available variables: ['abs440aer', 'abs500aer', 'abs550aer', 'abs550aercs', 'abs550bc', 'abs550dryaer', 'abs550dust', 'abs550oa', 'abs550so4', 'abs550ss', 'abs670aer', 'abs870aer', 'absc550aer', 'airmass', 'area', 'asy3Daer', 'bc5503Daer', 'cheaqpso4', 'chegpso4', 'chepso2', 'cl3D', 'clt', 'drybc', 'drydms', 'drydust', 'dryoa', 'dryso2', 'dryso4', 'dryss', 'ec550aer', 'ec550dryaer', 'emibc', 'emidms', 'emidust', 'emioa', 'emiso2', 'emiso4', 'emiss', 'hus', 'landf', 'loadbc', 'loaddms', 'loaddust', 'loadoa', 'loadso2', 'loadso4', 'loadss', 'mmraerh2o', 'mmrbc', 'mmrdu', 'mmroa', 'mmrso4', 'mmrss', 'od440aer', 'od440csaer', 'od500aer', 'od550aer', 'od550aerh2o', 'od550bc', 'od550csaer', 'od550dust', 'od550gt1aer', 'od550lt1aer', 'od550lt1dust', 'od550oa', 'od550so4', 'od550ss', 'od670aer', 'od870aer', 'od870csaer', 'orog', 'precip', 'pressure', 'ps', 'rlds', 'rlus', 'rlut', 'rlutcs', 'rsds', 'rsdscs', 'rsdt', 'rsus', 'rsut', 'sconcbc', 'sconcdms', 'sconcdust', 'sconcoa', 'sconcso2', 'sconcso4', 'sconcss', 'temp', 'vmrdms', 'vmrso2', 'wetbc', 'wetdms', 'wetdust', 'wetoa', 'wetso2', 'wetso4', 'wetss']
ANOTHER_MODEL_ID = 'CAM53-Oslo_7310_MG15CLM45_5feb2017IHK_53OSLO_PD_UNTUNED'
colocator.run(model_id=ANOTHER_MODEL_ID)
colocator.data.keys()
dict_keys(['TM5_AP3-CTRL2019', 'CAM53-Oslo_7310_MG15CLM45_5feb2017IHK_53OSLO_PD_UNTUNED'])
colocator.data[ANOTHER_MODEL_ID]['ang4487aer'].plot_scatter(marker='o', alpha=0.1, color='g');
_arr = col_data.data
mean_bias = ((_arr[1] - _arr[0])/_arr[0]).mean('time') * 100 #%
mean_bias
<xarray.DataArray 'od550aer' (station_name: 290)> array([ 3.27914642e+01, 4.41977593e+01, -2.38407751e+01, -2.20405471e+01, 4.87610882e+01, -1.09660243e+01, 6.09572635e+01, 6.95185320e+01, 5.72888842e+01, 2.02739190e+00, -5.72097325e+01, 1.47671090e+01, 4.29767724e+01, 4.29703426e+01, -3.04770598e+00, 3.25237489e+01, 3.04941723e+01, -3.50740249e+01, -2.67254451e+00, 7.03229780e+01, -1.73058152e+01, -2.54449063e+01, 1.17741062e+01, -3.34088968e+00, 5.11034734e+01, -5.49587609e+01, 3.12364125e+01, 1.41865023e+01, 9.79946795e+00, 4.29076415e+01, 2.00427413e+02, 3.11295196e+01, 1.25609668e+01, -2.99052611e+00, 3.79720939e+01, 4.11353824e+01, 3.36367237e+01, -1.87316612e+01, 5.71903871e+00, -3.12985326e+01, 9.64239738e+00, 3.87242518e+01, 3.28465450e+01, -1.34600998e+01, 1.13944697e+02, -5.74046714e+00, -2.21200918e+01, -1.68933975e+01, 2.14213839e+01, -1.57128817e+00, 1.63393518e+01, 3.03617771e+01, -3.66373152e+01, -2.29289206e+01, 3.80195940e+00, 2.28771411e+01, 5.02988951e+01, -8.47645363e+00, -2.92246798e-02, 1.99113391e+01, 2.85090981e+01, 1.02081703e+02, -3.32672024e+01, -2.61841440e+01, 1.02246696e+01, 3.35796759e+01, 2.47245469e+01, 3.40265035e+00, 4.58925360e+01, 4.16583949e+01, -2.69435038e+01, -1.98879423e+01, -2.25045093e+01, -2.61745328e+01, -2.19739435e+01, -3.29913089e+01, -2.35682266e+01, -2.66573040e+01, -3.18505908e+01, -1.84119126e+01, -2.15963517e+01, -3.09848055e+01, -1.06080074e+01, -1.25179840e+01, 8.36437743e+01, -1.72089120e+01, 1.10724291e+02, 1.86954451e+01, -3.52485746e+00, 2.26953688e+01, -1.31471950e+01, -2.58177183e+01, 3.19353929e+00, 1.67985095e+02, 2.15939248e+03, 2.48500492e+01, 2.09847766e+01, 2.98324464e+01, 3.56604135e+00, 1.34239937e+01, 8.50623273e+00, 4.48262894e+01, 2.34873722e+01, 5.06346266e+01, 2.74506461e+01, 7.61382986e+01, -3.28731649e+01, 4.07660799e+01, 2.27447667e+01, 9.10353067e+01, 1.21278828e+01, 3.72121519e+01, 2.83294548e+01, 9.78441851e+00, -4.73056445e+01, 2.79024380e+01, 5.35009477e+01, 5.91444655e+01, 4.79116057e+00, 7.44237999e+01, 2.68128184e+01, 3.43649449e+01, -1.62632855e+01, 1.30548675e+01, 6.61681390e+00, -3.00923781e+01, 2.23825834e+01, 2.76474665e+01, 4.34143410e+01, -2.69770526e+01, -5.00112156e-01, 8.53502032e+01, 2.87528003e+00, -4.53577189e+01, 2.41361076e+01, 4.31485335e+01, -2.39311000e+01, 1.25558526e+02, 4.14590368e+01, 3.16795086e+02, 2.92134127e+01, 1.80532386e+00, 1.70037355e+01, 1.62844145e+01, 4.42332748e+00, -1.34480446e+01, 8.70741225e+01, -2.57484589e+01, -1.49912499e+01, 2.98588810e+01, 1.59062183e+01, 1.03563362e+01, 2.72479298e+01, 3.76470196e+01, -3.95993626e+01, 1.21604190e+01, 5.86778380e+00, 3.10902325e+01, 1.05128068e+01, -4.00757249e+00, 4.05601341e+00, -3.14303075e+01, 8.17878429e+01, 4.30353716e+01, 6.44585924e+01, -1.27336460e+01, 4.54585568e+00, 2.35222980e+01, 4.67379278e+02, -8.97091250e+00, 1.80895809e+01, 5.12300032e+00, 3.14730287e+00, -2.68587539e+01, 2.66964067e+01, 4.73865279e+01, 8.04935946e+02, 4.17767451e+01, -4.21336847e+01, -9.07905949e-01, 7.66013358e+01, 8.37915206e+01, 2.13665125e+01, 1.85619169e+01, -1.89324207e+01, 3.32072539e+01, -1.75130819e+01, 2.18796414e+01, 1.97294485e+01, 1.16031417e+01, -1.80260862e+01, 1.06872806e+00, 2.50877329e+02, 1.86854932e-01, -6.29763143e+01, 7.34409702e+01, -2.42201841e+01, 2.04096214e+01, 2.84237510e+02, -2.83174167e+01, 2.75985008e-01, -1.77013012e+00, 9.33123068e+01, 8.96354615e+01, 1.16009139e+01, 5.89801647e+00, 3.53473407e+01, 2.40858232e+00, 5.26359199e-01, 1.39417708e+01, -9.18728962e+00, -2.89676009e+01, 2.11628891e+01, -4.27510894e+01, 5.47806817e+01, 1.11179355e+01, 1.60639161e+02, 5.36604379e+01, 1.41343458e+01, 7.29047268e+01, 1.16645319e+02, 1.07994446e+00, 2.86391474e+01, -6.17067746e+00, 3.15634790e+01, 9.71369792e+00, -4.73990236e+01, 2.01213515e+01, 1.50614483e+01, -2.92210408e+01, 8.26026117e+01, -9.81651091e+00, 2.18821079e+01, 4.22057093e+01, -1.68488604e+01, 3.47089193e+01, 3.79152838e+01, 7.39133554e+01, 3.63400929e+00, 6.77855757e+01, -2.33014879e+01, -4.99106302e+01, 4.25943052e+01, 3.12377925e+00, -2.27771852e+01, -1.20066272e+01, -2.66136810e+01, 7.31645234e+01, 1.52243559e+02, 3.19089141e+01, 2.96238395e+01, 8.33273265e+00, -1.91530812e+01, -1.87918140e+01, 3.24915367e+01, 4.61579198e+01, -4.67132468e+00, 1.50885510e+01, -1.13247846e+01, 7.83158893e+00, 4.29157881e+01, 3.62139244e+01, 2.62529907e+01, 4.16605716e+01, 1.65626661e+01, 4.09422433e+01, 2.31975416e+01, 1.07226551e+02, 1.58904995e+01, 1.37649886e+01, 5.04940170e+00, -2.76175068e+01, 7.44272366e+00, 5.05170565e+01, 1.01163561e+02, 2.09883399e+01, 1.00184819e+01, -2.85436142e+01, 2.31967797e+01, 4.90043152e+00, 1.17823575e+02, 6.87338294e+01, -2.43874895e+01, 7.69502307e+00, 4.11403093e+01, 1.09082297e+02, 2.00864470e+01, 1.99950303e+01, 3.62194452e+01, 1.41876019e+01]) Coordinates: var_name <U8 'od550aer' var_units <U1 '1' * station_name (station_name) <U19 'ARM_Darwin' ... 'Zinder_Airport' latitude (station_name) float64 -12.43 37.97 15.35 ... 62.45 13.78 longitude (station_name) float64 130.9 23.72 -1.479 ... -114.4 8.99 altitude (station_name) float64 29.9 130.0 305.0 ... 300.0 220.8 456.0 data_source <U22 'AeronetSunV3Lev2.daily' ts_type_src <U7 'daily'
ax = pya.plot.mapping.init_map()
ax = pya.plot.mapping.init_map()
_sc = ax.scatter(mean_bias.longitude, mean_bias.latitude, marker='o', c=mean_bias.data,
cmap='bwr',
vmin=-100, vmax=100, s=80)
cb = plt.colorbar(_sc)
cb.ax.set_ylabel('Normalised mean bias [%]');
https://aerocom-evaluation.met.no/overall.php?project=aerocom&exp=PIII-optics2019