Sandbox Metadata Document

This document aims to provide an overview of the different data sets available in the Open Data Cube Sandbox. It will provide details on which products are available, what regions are indexed for each product, and the time series available within that extent.

For information on how to get started with the Datacube, please see the read_me_first.ipynb available here.

Datacube Information

In [1]:
from IPython.core.display import display, HTML
import pandas as pd

import datacube
dc = datacube.Datacube()
dc.list_products()
Out[1]:
name description creation_time lat platform lon format time instrument product_type label crs resolution tile_size spatial_dimensions
id
15 alos2_palsar_AMA_ingest ALOS-2 PALSAR mosaic tiles generated for use i... None None ALOS_2 None GeoTIFF None PALSAR gamma0 None NaN NaN NaN NaN
6 ls5_level1_usgs Landsat 5 USGS Level 1 Collection-1 OLI-TIRS None None LANDSAT_5 None GeoTiff None TM L1TP None NaN NaN NaN NaN
19 ls5_usgs_sr_scene Landsat 5 USGS Collection 1 Level2 Surface Ref... None None LANDSAT_5 None GeoTiff None TM LEVEL2_USGS None NaN NaN NaN NaN
17 ls7_collection1_AMA_ingest Landsat 7 USGS Collection 1 Higher Level SR sc... None None LANDSAT_7 None GeoTiff None ETM LEDAPS None NaN NaN NaN NaN
5 ls7_level1_usgs Landsat 7 USGS Level 1 Collection-1 OLI-TIRS None None LANDSAT_7 None GeoTiff None ETM L1TP None NaN NaN NaN NaN
18 ls7_usgs_sr_scene Landsat 7 USGS Collection 1 Level2 Surface Ref... None None LANDSAT_7 None GeoTiff None ETM LEVEL2_USGS None NaN NaN NaN NaN
16 ls8_collection1_AMA_ingest Landsat 8 USGS Collection 1 Higher Level SR sc... None None LANDSAT_8 None GeoTIFF None OLI_TIRS LaSRC None NaN NaN NaN NaN
7 ls8_l1_pc_usgs Landsat 8 USGS Level 1 Pre-Collection OLI-TIRS None None LANDSAT_8 None GeoTiff None OLI_TIRS L1T None NaN NaN NaN NaN
4 ls8_level1_usgs Landsat 8 USGS Level 1 Collection-1 OLI-TIRS None None LANDSAT_8 None GeoTiff None OLI_TIRS L1TP None NaN NaN NaN NaN
21 ls8_usgs_sr_scene Landsat 8 USGS Collection 1 Higher Level SR sc... None None LANDSAT_8 None GeoTiff None OLI_TIRS LEVEL2_USGS None NaN NaN NaN NaN
13 s1_gamma0_scene Sentinel-1A/B SAR Gamma0 scenes, processed to ... None None SENTINEL_1 None GeoTIFF None SAR gamma0 None NaN NaN NaN NaN

alos2_palsar_AMA_ingest

Caqueta
Time: 2015-01-01 and 2016-01-01
latitude: ( 0, 3.2)
longitude: (-76.26666668, -73.6)

Vietnam
Time: 2015-01-01 and 2016-01-01
latitude: ( 7.46666667, 12.8)
longitude: (103.46666668, 110.4)


ls5_level1_usgs

No Data Indexed


ls5_usgs_sr_scene

Time: 1984-05-26 to 2011-11-13
y: (-587700.0, 2187300.0)
x: ( 68100.0, 932400.0)


ls7_collection1_AMA_ingest

Caqueta
Time: 1999-08-21 to 2018-03-25
latitude: ( 0.000134747292617865, 1.077843593651382)
longitude: (-74.91935994831539, -73.30266193148462)
resolution: ( -0.000269494585236, 0.000269494585236)

Lake Baringo, Kenya
Time: 2005-01-08 to 2016-12-24
latitude: ( 0.4997747685, 0.7495947795)
longitude: (35.9742163305, 36.473586859499996)
resolution: (-0.000269493, 0.000269493)


ls7_level1_usgs

No Data Indexed


ls7_usgs_sr_scene

Time: 1999-07-08 to 2018-11-02
y: (-586200.0, 2185200.0)
x: ( 70800.0, 935100.0)


ls8_collection1_AMA_ingest

Caqueta
Time: 2013-04-13 to 2018-03-26
latitude: ( 0.000134747292617865, 1.077843593651382)
longitude: (-74.91935994831539, -73.30266193148462)
resolution: ( -0.000269494585236, 0.000269494585236)

Vietnam
Time: 2014-01-14 to 2016-12-21
latitude: ( 10.513927001104687, 12.611133863411238)
longitude: (106.79005909290998, 108.91906631627438)
resolution: ( -0.000269494585236, 0.000269494585236)


ls8_l1_pc_usgs

No Data Indexed


ls8_level1_usgs

Global


ls8_usgs_sr_scene

Time: 2013-03-21 to 2018-11-16
y: (-596700.0, 2193300.0)
x: ( 82500.0, 930300.0)


s1_gamma0_scene

Time: 2015-05-12 to 2018-07-20

Caqueta
latitude: ( 1.00018083, 2.)
longitude: (-75., -74.)

Samoa
latitude: ( -13.99981917, -13.)
longitude: (-171., -173.)

Plotting Indicative Data Extents

The following code generates an overview of the approximte extents of the product selected on line 1, shown in red. The yellow base image shows the extents of the global ls8_level1_usgs product.

In [2]:
product = 'ls8_collection1_AMA_ingest'

import matplotlib.pyplot as plt
from pyproj import Proj, transform

import datacube
dc = datacube.Datacube()

outProj = Proj(init='epsg:4326')

x_bounds = []
y_bounds = []
unique_bounds=set()
bounds=[]
times=[]
tiles = dc.find_datasets(product=product)
if max(tiles[0].bounds) > 180: XY = True
else: XY = False

for tile in tiles:
    if XY:
        inProj = Proj(init=tile.crs)
        l_lon,t_lat = transform(inProj,outProj,tile.bounds.left,tile.bounds.top)
        r_lon,b_lat = transform(inProj,outProj,tile.bounds.right,tile.bounds.bottom)
        x_bounds = [l_lon, r_lon, r_lon, l_lon]
        y_bounds = [t_lat, t_lat, b_lat, b_lat]
    
    else:
        x_bounds = [tile.bounds.left, tile.bounds.right, tile.bounds.right, tile.bounds.left]
        y_bounds = [tile.bounds.top, tile.bounds.top, tile.bounds.bottom, tile.bounds.bottom]
        
    times.append(tile.time)
    bounds_key = "".join(str(x) for x in x_bounds) + "".join(str(y) for y in y_bounds)
    if bounds_key not in unique_bounds:
        bounds.append([x_bounds, y_bounds, tile.time])

%matplotlib inline
fig, ax = plt.subplots(1, figsize = (18,9))
for i in bounds:
    ax.fill(i[0], i[1], c='red')
ax.axis('equal')

plt.title(product)
ax.set_xlabel('Longitude')
ax.set_ylabel('Latitude')
basemap = plt.imread('../images/indexed_areas.png')
ax.imshow(basemap, extent=[-180,180,-90,90])
ax.set_xlim(-180,180)
ax.set_ylim(-90, 90)
plt.show()
print('Time:', min(times[-1]).strftime('%Y-%m-%d'), 'to', max(times[0]).strftime('%Y-%m-%d'))
Time: 2013-04-13 to 2018-03-26

System Information

In [3]:
import sys
print('Python Version', sys.version)
Python Version 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0]
In [4]:
!pip3 show datacube xarray numpy pandas gdal matplotlib 
Name: datacube
Version: 1.6.1+179.gc4ed2929.dirty
Summary: An analysis environment for satellite and other earth observation data
Home-page: https://github.com/opendatacube/datacube-core
Author: Open Data Cube
Author-email: None
License: Apache License 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: affine, cachetools, click, cloudpickle, dask, gdal, jsonschema, netcdf4, numpy, psycopg2, pypeg2, python-dateutil, pyyaml, rasterio, singledispatch, sqlalchemy, toolz, xarray
---
Name: xarray
Version: 0.10.7
Summary: N-D labeled arrays and datasets in Python
Home-page: https://github.com/pydata/xarray
Author: xarray Developers
Author-email: [email protected]
License: Apache
Location: /usr/local/lib/python3.6/dist-packages
Requires: pandas, numpy
---
Name: numpy
Version: 1.15.4
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /home/jovyan/.local/lib/python3.6/site-packages
Requires: 
---
Name: pandas
Version: 0.23.4
Summary: Powerful data structures for data analysis, time series, and statistics
Home-page: http://pandas.pydata.org
Author: None
Author-email: None
License: BSD
Location: /home/jovyan/.local/lib/python3.6/site-packages
Requires: python-dateutil, pytz, numpy
---
Name: GDAL
Version: 2.3.2
Summary: GDAL: Geospatial Data Abstraction Library
Home-page: http://www.gdal.org
Author: Howard Butler
Author-email: [email protected]
License: MIT
Location: /usr/lib/python3/dist-packages
Requires: 
---
Name: matplotlib
Version: 3.0.2
Summary: Python plotting package
Home-page: http://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email: [email protected]
License: BSD
Location: /home/jovyan/.local/lib/python3.6/site-packages
Requires: pyparsing, cycler, python-dateutil, numpy, kiwisolver

Extent Query Functions

In [5]:
#Generates text extents of data
product = 's1_gamma0_scene'

import datacube
dc = datacube.Datacube()
import numpy as np

left = np.array([])
bottom = np.array([])
right = np.array([])
top = np.array([])
times = np.array([])

tiles = dc.find_datasets(product=product)
for tile in tiles: #Each tile (unique time for each)
    left = np.append(left, tile.bounds[0])
    right = np.append(right, tile.bounds[2])
    top = np.append(top, tile.bounds[3])
    bottom = np.append(bottom, tile.bounds[1])
    times = np.append(times, tile.time)
    #corners = np.append(corners, [])
print('Max Extent')
print('``Time:', min(times).strftime('%Y-%m-%d'), 'to',  max(times).strftime('%Y-%m-%d')+'``  ')
print('``latitude:  ('+str(min(np.unique(bottom)))+', '+str(max(np.unique(top)))+')``  ')
print('``longitude: ('+str(min(np.unique(left)))+', '+str(max(np.unique(right)))+')``  ')

print('Discrete Extents')
print('``Time:', min(times).strftime('%Y-%m-%d'), 'to',  max(times).strftime('%Y-%m-%d')+'``  ')
print('``latitude:  ('+str(np.unique(bottom))+', '+str(np.unique(top))+')``  ')
print('``longitude: ('+str(np.unique(left))+', '+str(np.unique(right))+')``  ')
Max Extent
``Time: 2015-05-12 to 2018-07-20``  
``latitude:  (-13.9998191681736, 2.0)``  
``longitude: (-173.0, -74.0)``  
Discrete Extents
``Time: 2015-05-12 to 2018-07-20``  
``latitude:  ([-13.99981917   1.00018083], [-13.   2.])``  
``longitude: ([-173. -172.  -75.], [-172. -171.  -74.])``  
In [6]:
pd.options.display.max_rows = 200
dc.list_measurements()
Out[6]:
aliases dtype flags_definition name nodata units
product measurement
alos2_palsar_AMA_ingest hh NaN int16 NaN hh 0 DN
hv NaN int16 NaN hv 0 DN
date NaN int16 NaN date 0 1
incidence_angle NaN uint8 NaN incidence_angle 0 1
mask NaN uint8 {'cloud_confidence': {'bits': [0, 1, 2, 3, 4, ... mask 0 1
ls5_level1_usgs blue [band_1, blue] int16 NaN blue -9999 1
green [band_2, green] int16 NaN green -9999 1
red [band_3, red] int16 NaN red -9999 1
nir [band_4, nir] int16 NaN nir -9999 1
swir1 [band_5, swir1] int16 NaN swir1 -9999 1
swir2 [band_7, swir2] int16 NaN swir2 -9999 1
quality [QUALITY, quality] int16 {'cloud': {'bits': [4], 'values': {'0': False,... quality 0 1
ls5_usgs_sr_scene blue [band_1, sr_band1] int16 NaN blue -9999 reflectance
green [band_2, sr_band2] int16 NaN green -9999 reflectance
red [band_3, sr_band3] int16 NaN red -9999 reflectance
nir [band_4, sr_band4] int16 NaN nir -9999 reflectance
swir1 [band_5, sr_band5] int16 NaN swir1 -9999 reflectance
swir2 [band_7, sr_band7] int16 NaN swir2 -9999 reflectance
sr_atmos_opacity [atmos_op] uint8 NaN sr_atmos_opacity 0 1
pixel_qa [pixel_qa] uint16 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... pixel_qa 1 bit_index
radsat_qa [radsat_qa] uint8 {'radsat_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7... radsat_qa 1 bit_index
sr_cloud_qa [pixel_qa] uint16 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... sr_cloud_qa 1 bit_index
ls7_collection1_AMA_ingest blue NaN int16 NaN blue -9999 reflectance
green NaN int16 NaN green -9999 reflectance
red NaN int16 NaN red -9999 reflectance
nir NaN int16 NaN nir -9999 reflectance
swir1 NaN int16 NaN swir1 -9999 reflectance
swir2 NaN int16 NaN swir2 -9999 reflectance
atmos_opacity NaN uint8 NaN atmos_opacity 0 unitless
pixel_qa NaN int32 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... pixel_qa 1 bit_index
radsat_qa NaN uint8 {'radsat_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7... radsat_qa 1 bit_index
cloud_qa NaN uint8 {'cloud_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7]... cloud_qa 0 bit_index
ls7_level1_usgs blue [band_1, blue] int16 NaN blue -9999 1
green [band_2, green] int16 NaN green -9999 1
red [band_3, red] int16 NaN red -9999 1
nir [band_4, nir] int16 NaN nir -9999 1
swir1 [band_5, swir1] int16 NaN swir1 -9999 1
swir2 [band_7, swir2] int16 NaN swir2 -9999 1
quality [QUALITY, quality] int16 {'cloud': {'bits': [4], 'values': {'0': False,... quality 0 1
ls7_usgs_sr_scene blue [band_1, sr_band1] int16 NaN blue -9999 reflectance
green [band_2, sr_band2] int16 NaN green -9999 reflectance
red [band_3, sr_band3] int16 NaN red -9999 reflectance
nir [band_4, sr_band4] int16 NaN nir -9999 reflectance
swir1 [band_5, sr_band5] int16 NaN swir1 -9999 reflectance
swir2 [band_7, sr_band7] int16 NaN swir2 -9999 reflectance
sr_atmos_opacity [atmos_op] uint8 NaN sr_atmos_opacity 0 1
pixel_qa [pixel_qa] uint16 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... pixel_qa 1 bit_index
radsat_qa [radsat_qa] uint8 {'radsat_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7... radsat_qa 1 bit_index
sr_cloud_qa [cloud_qa] uint8 {'cloud_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7]... sr_cloud_qa 0 bit_index
ls8_collection1_AMA_ingest coastal_aerosol NaN int16 NaN coastal_aerosol -9999 reflectance
blue NaN int16 NaN blue -9999 reflectance
green NaN int16 NaN green -9999 reflectance
red NaN int16 NaN red -9999 reflectance
nir NaN int16 NaN nir -9999 reflectance
swir1 NaN int16 NaN swir1 -9999 reflectance
swir2 NaN int16 NaN swir2 -9999 reflectance
pixel_qa NaN int32 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... pixel_qa 1 bit_index
aerosol_qa NaN int16 {'aerosol_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, ... aerosol_qa 0 bit_index
radsat_qa NaN int32 {'radsat_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7... radsat_qa 1 bit_index
ls8_l1_pc_usgs coastal_aerosol [band_1, coastal_aerosol] int16 NaN coastal_aerosol -9999 1
blue [band_2, blue] int16 NaN blue -9999 1
green [band_3, green] int16 NaN green -9999 1
red [band_4, red] int16 NaN red -9999 1
nir [band_5, nir] int16 NaN nir -9999 1
swir1 [band_6, swir1] int16 NaN swir1 -9999 1
swir2 [band_7, swir2] int16 NaN swir2 -9999 1
panchromatic [band_8, panchromatic] int16 NaN panchromatic -9999 1
cirrus [band_9, cirrus] int16 NaN cirrus -9999 1
lwir1 [band_10, lwir1] int16 NaN lwir1 -9999 1
lwir2 [band_11, lwir2] int16 NaN lwir2 -9999 1
quality [QUALITY, quality] int16 {'water': {'bits': [4, 5], 'values': {'0': Fal... quality 0 1
ls8_level1_usgs coastal_aerosol [band_1, coastal_aerosol] int16 NaN coastal_aerosol -9999 1
blue [band_2, blue] int16 NaN blue -9999 1
green [band_3, green] int16 NaN green -9999 1
red [band_4, red] int16 NaN red -9999 1
nir [band_5, nir] int16 NaN nir -9999 1
swir1 [band_6, swir1] int16 NaN swir1 -9999 1
swir2 [band_7, swir2] int16 NaN swir2 -9999 1
panchromatic [band_8, panchromatic] int16 NaN panchromatic -9999 1
cirrus [band_9, cirrus] int16 NaN cirrus -9999 1
lwir1 [band_10, lwir1] int16 NaN lwir1 -9999 1
lwir2 [band_11, lwir2] int16 NaN lwir2 -9999 1
quality [QUALITY, quality] int16 {'cloud': {'bits': [4], 'values': {'0': False,... quality 0 1
ls8_usgs_sr_scene coastal_aerosol [band_1, sr_band1] int16 NaN coastal_aerosol -9999 reflectance
blue [band_2, sr_band2] int16 NaN blue -9999 reflectance
green [band_3, sr_band3] int16 NaN green -9999 reflectance
red [band_4, sr_band4] int16 NaN red -9999 reflectance
nir [band_5, sr_band5] int16 NaN nir -9999 reflectance
swir1 [band_6, sr_band6] int16 NaN swir1 -9999 reflectance
swir2 [band_7, sr_band7] int16 NaN swir2 -9999 reflectance
pixel_qa [pixel_qa] uint16 {'snow': {'bits': 4, 'values': {'0': 'no_snow'... pixel_qa 1 bit_index
sr_aerosol [sr_aerosol_qa, aerosol_qa, aerosol] uint8 {'aerosol_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, ... sr_aerosol 0 bit_index
radsat_qa [radsat_qa] uint16 {'radsat_qa': {'bits': [0, 1, 2, 3, 4, 5, 6, 7... radsat_qa 1 bit_index
s1_gamma0_scene vh NaN float32 NaN vh 0 dB
vv NaN float32 NaN vv 0 dB