This 1 arc second (~30m) resolution terrain data as 32-bit cloud-optimized geotiff in a public AWS bucket. Each file is a 1x1 degree square.
import hvplot.xarray
import fsspec
import os
import rioxarray
Use GDAL settings from COG Best Practices
os.environ['GDAL_DISABLE_READDIR_ON_OPEN']='EMPTY_DIR' #This is KEY! otherwise we send a bunch of HTTP GET requests to test for common sidecar metadata
os.environ['AWS_NO_SIGN_REQUEST']='YES' #Since this is a public bucket, we don't need authentication
os.environ['GDAL_MAX_RAW_BLOCK_CACHE_SIZE']='200000000' #200MB: Want this to be greater than size of uncompressed raster to overcome a 10 MB limit in the GeoTIFF driver for range request merging.
os.environ['GDAL_SWATH_SIZE']='200000000' #also increase this if increasing MAX_RAW_BLOCK_CACHE_SIZE
os.environ['VSI_CURL_CACHE_SIZE']='200000000' #also increase this if increasing MAX_RAW_BLOCK_CACHE_SIZE
Explore a sample terrain square (the 1x1 degree square that contains most of Cape Cod)
url = 'https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1/TIFF/n42w071/USGS_1_n42w071.tif'
da = rioxarray.open_rasterio(url, masked=True).squeeze('band')
da
<xarray.DataArray (y: 3612, x: 3612)>
[13046544 values with dtype=float64]
Coordinates:
band int64 1
* y (y) float64 42.0 42.0 42.0 42.0 42.0 ... 41.0 41.0 41.0 41.0
* x (x) float64 -71.0 -71.0 -71.0 -71.0 ... -70.0 -70.0 -70.0 -70.0
spatial_ref int64 0
Attributes:
LAYER_TYPE: athematic
RepresentationType: ATHEMATIC
scale_factor: 1.0
add_offset: 0.0
long_name: Layer_1
grid_mapping: spatial_ref[13046544 values with dtype=float64]
array(1)
array([42.001528, 42.00125 , 42.000972, ..., 40.999028, 40.99875 , 40.998472])
array([-71.001528, -71.00125 , -71.000972, ..., -69.999028, -69.99875 ,
-69.998472])array(0)
Mask out the ocean
da = da.where(da>0)
Visualize with hvplot from the holoviz.org tool suite
da.hvplot.image(x='x', y='y', geo=True, rasterize=True,
frame_width=500, cmap='rainbow', tiles='ESRI')
fs = fsspec.filesystem('s3', anon=True)
flist = fs.glob('prd-tnm/StagedProducts/Elevation/1/TIFF/')
flist[:5]
['prd-tnm/StagedProducts/Elevation/1/TIFF/n06e162', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n06e163', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n07e134', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n07e151', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n07e152']
flist[-5:]
['prd-tnm/StagedProducts/Elevation/1/TIFF/n84w076', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n84w077', 'prd-tnm/StagedProducts/Elevation/1/TIFF/n84w078', 'prd-tnm/StagedProducts/Elevation/1/TIFF/s14w170', 'prd-tnm/StagedProducts/Elevation/1/TIFF/s14w171']