Accessing SnowEx Data at the NSIDC DAAC
Contents
Accessing SnowEx Data at the NSIDC DAAC¶
2022 SnowEx Hackweek
Authors: Amy Steiker, Gail Reckase NSIDC DAAC, CIRES, University of Colorado
This tutorial provides a brief overview of the resources provided by the NASA National Snow and Ice Data Center Distributed Active Archive Center, or NSIDC DAAC, and demonstrates how to discover and access SnowEx data programmatically.
Learning Objectives:¶
Identify resources available at the NSIDC DAAC to help you work with SnowEx data.
Search and discover file size and coverage of SnowEx data over a time and geojson region of interest.
Set up Earthdata Login authentication needed to access NASA Earthdata.
Download SnowEx data programmatically using the NSIDC DAAC Application Programming Interface (API).
___¶
Explore snow products and resources¶
NSIDC introduction¶
The National Snow and Ice Data Center, located in Boulder, Colorado, provides over 1300 data sets covering the Earth’s cryosphere and more, all of which are available to the public free of charge. Beyond providing these data, NSIDC creates tools for data access, supports data users, performs scientific research, and educates the public about the cryosphere.
The NASA National Snow and Ice Data Center Distributed Active Archive Center (NSIDC DAAC) provides access to over 750 data products (over 2 PB of data and growing!) in support of cryospheric research, global change detection, and water resource management. NSIDC is one of 12 DAACs as part of NASA’s EOSDIS, or Earth Observing System Data and Information System, and is also the largest program within the National Snow and Ice Data Center, as part of the University of Colorado’s Cooperative Institute for Research in Environmental Sciences.
Select Data Resources¶
-
Jupyter Notebook tutorials providing guidance on working with various NSIDC DAAC data products, including how to access, subset, transform, and visualize data.
-
Search NSIDC snow data
NSIDC Data Update Announcements
News and tips for data users
-
Search and access data across the NASA Earthdata
-
Interactive interface for browsing full-resolution, global, daily satellite images
Snow Today¶
Snow Today, a collaboration with the University of Colorado’s Institute of Alpine and Arctic Research (INSTAAR), provides near-real-time snow analysis for the western United States and regular reports on conditions during the winter season. Snow Today is funded by NASA Hydrological Sciences Program and utilizes data from the Moderate Resolution Imaging Spectroradiometer (MODIS) instrument and snow station data from the Snow Telemetry (SNOTEL) network by the Natural Resources Conservation Service (NRCS), United States Department of Agriculture (USDA) and the California Department of Water Resources: www.wcc.nrcs.usda.gov/snow.
NSIDC’s SnowEx pages¶
-
We will be using the ‘SnowEx17 Ground Penetrating Radar, Version 2’ dataset in the following steps: https://doi.org/10.5067/G21LGCNLFSC5
Other relevant snow products:¶
Near-Real-Time SSM/I-SSMIS EASE-Grid Daily Global Ice Concentration and Snow Extent (NISE), Version 5: https://doi.org/10.5067/3KB2JPLFPK3R
Import Packages¶
Get started by importing packages needed to run the following code blocks. Shapely allows for the manipulation of geometric objects. Geopandas is used to assist in working with geospatial data.
import os
import geopandas as gpd
from shapely.geometry import Polygon, mapping
from shapely.geometry.polygon import orient
import pandas as pd
import requests
import json
import pprint
import getpass
import netrc
from platform import system
from getpass import getpass
from urllib import request
from http.cookiejar import CookieJar
from os.path import join, expanduser
import requests
from xml.etree import ElementTree as ET
import time
import zipfile
import io
import shutil
Data Discovery¶
Start by identifying your study area and exploring coincident data over the same time and area.
Identify area and time of interest¶
Since our focus is on the Grand Mesa study site of the NASA SnowEx campaign, we’ll use that area to search for coincident data across other data products. From the SnowEx20 Grand Mesa IOP BSU 1 GHz Multi-polarization GPR CMP Snow Water Equivalent, Version 1 landing page, you can find the rectangular spatial coverage under the Overview tab, or you can draw a polygon over your area of interest in the map under the Download Data tab and export the shape as a geojson file using the Export Polygon icon shown below. An example polygon geojson file is provided in the /Data folder of this repository.
Create polygon coordinate string¶
Read in the geojson file as a GeoDataFrame object and simplify and reorder using the shapely package. This will be converted back to a dictionary to be applied as our polygon search parameter.
polygon_filepath = str(os.getcwd() + '/Data/nsidc-polygon.json') # Note: A shapefile or other vector-based spatial data format could be substituted here.
gdf = gpd.read_file(polygon_filepath) #Return a GeoDataFrame object
# Simplify polygon for complex shapes in order to pass a reasonable request length to CMR. The larger the tolerance value, the more simplified the polygon.
# Orient counter-clockwise: CMR polygon points need to be provided in counter-clockwise order. The last point should match the first point to close the polygon.
poly = orient(gdf.simplify(0.05, preserve_topology=False).loc[0],sign=1.0)
#Format dictionary to polygon coordinate pairs for CMR polygon filtering
polygon = ','.join([str(c) for xy in zip(*poly.exterior.coords.xy) for c in xy])
print('Polygon coordinates to be used in search:', polygon)
poly
Polygon coordinates to be used in search: -108.2352445938561,38.98556907427165,-107.85284607930835,38.978765032966244,-107.85494925720668,39.10596902171742,-108.22772795408136,39.11294532581687,-108.2352445938561,38.98556907427165
Alternatively: Specify bounding box region of interest¶
Instead of using a vector shape file to specify a region of interest, you can simply use a bounding box. The following cell is commented out below, which can be used instead of the polygon search parameter.
# bounds = poly.bounds # Get polygon bounds to be used as bounding box input
# bounding_box = ','.join(map(str, list(bounds))) # Bounding Box spatial parameter in decimal degree 'W,S,E,N' format.
# print(bounding_box)
Set time range¶
This is an optional parameter; set this to specify a time range of interest. In this case we’ll just select all of 2017 to ensure that we receive all files within this data set campaign.
temporal = '2020-01-01T00:00:00Z,2020-12-31T23:59:59Z' # Set temporal range
Create data parameter dictionary¶
Create a dictionary with the data set shortname and version, as well as the temporal range and polygonal area of interest. Data set shortnames, or IDs, as well as version numbers, are located at the top of every NSIDC landing page.
param_dict = {
'short_name': 'SNEX20_BSU_CMP_SWE',
'version': '1',
'polygon': polygon,
# 'bounding_box': bounding_box, #optional alternative to polygon search parameter; if using, remove or comment out polygon search parameter
'temporal':temporal,
}
Determine how many files exist over this time and area of interest, as well as the average size and total volume of those files¶
We will use the granule_info function to query metadata about each data set and associated files using the Common Metadata Repository (CMR), which is a high-performance, high-quality, continuously evolving metadata system that catalogs Earth Science data and associated service metadata records. Note that not all NSIDC data can be searched at the file level using CMR, particularly those outside of the NASA DAAC program.
def search_granules(search_parameters, geojson=None, output_format="json"):
"""
Performs a granule search with token authentication for restricted results
:search_parameters: dictionary of CMR search parameters
:token: CMR token needed for restricted search
:geojson: filepath to GeoJSON file for spatial search
:output_format: select format for results https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#supported-result-formats
:returns: if hits is greater than 0, search results are returned in chosen output_format, otherwise returns None.
"""
search_url = "https://cmr.earthdata.nasa.gov/search/granules"
if geojson:
files = {"shapefile": (geojson, open(geojson, "r"), "application/geo+json")}
else:
files = None
parameters = {
"scroll": "true",
"page_size": 100,
}
try:
response = requests.post(f"{search_url}.{output_format}", params=parameters, data=search_parameters, files=files)
response.raise_for_status()
except HTTPError as http_err:
print(f"HTTP Error: {http_error}")
except Exception as err:
print(f"Error: {err}")
hits = int(response.headers['CMR-Hits'])
if hits > 0:
print(f"Found {hits} files")
results = json.loads(response.content)
granules = []
granules.extend(results['feed']['entry'])
granule_sizes = [float(granule['granule_size']) for granule in granules]
print(f"The total size of all files is {sum(granule_sizes):.2f} MB")
return response.json()
else:
print("Found no hits")
return
search_granules(param_dict)
Found 2 files
The total size of all files is 1.01 MB
{'feed': {'updated': '2022-12-09T16:21:32.170Z',
'id': 'https://cmr.earthdata.nasa.gov:443/search/granules.json?short_name=SNEX20_BSU_CMP_SWE&version=1&polygon=-108.2352445938561%2C38.98556907427165%2C-107.85284607930835%2C38.978765032966244%2C-107.85494925720668%2C39.10596902171742%2C-108.22772795408136%2C39.11294532581687%2C-108.2352445938561%2C38.98556907427165&temporal=2020-01-01T00%3A00%3A00Z%2C2020-12-31T23%3A59%3A59Z',
'title': 'ECHO granule metadata',
'entry': [{'producer_granule_id': 'SNEX20_BSU_CMP_SWE_01312020',
'time_start': '2020-01-31T00:00:01.000Z',
'updated': '2021-01-29T12:44:24.929Z',
'dataset_id': 'SnowEx20 Grand Mesa IOP BSU 1 GHz Multi-polarization GPR CMP Snow Water Equivalent V001',
'points': ['39.02 -108.19'],
'data_center': 'NSIDC_ECS',
'title': 'SC:SNEX20_BSU_CMP_SWE.001:200101549',
'coordinate_system': 'GEODETIC',
'day_night_flag': 'UNSPECIFIED',
'time_end': '2020-01-31T23:59:59.000Z',
'id': 'G1998493977-NSIDC_ECS',
'original_format': 'ECHO10',
'granule_size': '0.508781',
'browse_flag': False,
'collection_concept_id': 'C1998359549-NSIDC_ECS',
'online_access_flag': True,
'links': [{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP1_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP1_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP2_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP2_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP3_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020_CMP3_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',
'type': 'text/xml',
'title': '(METADATA)',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.01.31/SNEX20_BSU_CMP_SWE_01312020.xml'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1998359549-NSIDC_ECS&pg[0][gsk]=-start_date&q=SNEX20_BSU_CMP_SWE&m=38.35552775499695!-113.57226562500001!5!1!0!0%2C2&tl=1596643427!4!!'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_BSU_CMP_SWE.001/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://nsidc.org/data/data-access-tool/SNEX20_BSU_CMP_SWE/versions/1/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1998359549-NSIDC_ECS&pg[0][gsk]=-start_date&q=SNEX20_BSU_CMP_SWE&m=38.35552775499695!-113.57226562500001!5!1!0!0%2C2&tl=1596643427!4!!'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_BSU_CMP_SWE.001/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://nsidc.org/data/data-access-tool/SNEX20_BSU_CMP_SWE/versions/1/'},
{'inherited': True,
'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',
'hreflang': 'en-US',
'href': 'https://doi.org/10.5067/SOFEX3867ECJ'},
{'inherited': True,
'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#',
'hreflang': 'en-US',
'href': 'https://doi.org/10.5067/SOFEX3867ECJ'}]},
{'producer_granule_id': 'SNEX20_BSU_CMP_SWE_02012020',
'time_start': '2020-02-01T00:00:01.000Z',
'updated': '2021-01-29T12:44:24.930Z',
'dataset_id': 'SnowEx20 Grand Mesa IOP BSU 1 GHz Multi-polarization GPR CMP Snow Water Equivalent V001',
'points': ['39.02 -108.19'],
'data_center': 'NSIDC_ECS',
'title': 'SC:SNEX20_BSU_CMP_SWE.001:200101547',
'coordinate_system': 'GEODETIC',
'day_night_flag': 'UNSPECIFIED',
'time_end': '2020-02-01T23:59:59.000Z',
'id': 'G1998494296-NSIDC_ECS',
'original_format': 'ECHO10',
'granule_size': '0.503572',
'browse_flag': False,
'collection_concept_id': 'C1998359549-NSIDC_ECS',
'online_access_flag': True,
'links': [{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP1_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP1_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP2_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP2_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP3_HH.csv'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'type': 'text/plain',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020_CMP3_HH.png'},
{'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',
'type': 'text/xml',
'title': '(METADATA)',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/DP6/SNOWEX/SNEX20_BSU_CMP_SWE.001/2020.02.01/SNEX20_BSU_CMP_SWE_02012020.xml'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1998359549-NSIDC_ECS&pg[0][gsk]=-start_date&q=SNEX20_BSU_CMP_SWE&m=38.35552775499695!-113.57226562500001!5!1!0!0%2C2&tl=1596643427!4!!'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_BSU_CMP_SWE.001/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://nsidc.org/data/data-access-tool/SNEX20_BSU_CMP_SWE/versions/1/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://search.earthdata.nasa.gov/search/granules?p=C1998359549-NSIDC_ECS&pg[0][gsk]=-start_date&q=SNEX20_BSU_CMP_SWE&m=38.35552775499695!-113.57226562500001!5!1!0!0%2C2&tl=1596643427!4!!'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://n5eil01u.ecs.nsidc.org/SNOWEX/SNEX20_BSU_CMP_SWE.001/'},
{'inherited': True,
'length': '0.0KB',
'rel': 'http://esipfed.org/ns/fedsearch/1.1/data#',
'hreflang': 'en-US',
'href': 'https://nsidc.org/data/data-access-tool/SNEX20_BSU_CMP_SWE/versions/1/'},
{'inherited': True,
'rel': 'http://esipfed.org/ns/fedsearch/1.1/metadata#',
'hreflang': 'en-US',
'href': 'https://doi.org/10.5067/SOFEX3867ECJ'},
{'inherited': True,
'rel': 'http://esipfed.org/ns/fedsearch/1.1/documentation#',
'hreflang': 'en-US',
'href': 'https://doi.org/10.5067/SOFEX3867ECJ'}]}]}}
Data Access¶
Option 1: Python script from the data set landing page¶
There is a python script provided under the “Download Data” tab of each NSIDC DAAC data set landing page.
Earthdata Login authentication setup notes¶
All data with the NASA Earthdata system are freely available to the public, but requires an Earthdata Login profile to access. If you have not already done so, visit http://urs.earthdata.nasa.gov to register (it just takes a minute to set up).
Create a
.netrcfile within the JupyterHub environment. This is a hidden file typically stored in your home directory that contains your Earthdata login username and password. This is a secure and easy way to ensure that any data download requests are authenticated against your profile. You can set up your.netrcwithin the JupyterHub environment as copied from the preliminary work article here:
Run the following commands on the JupyterHub in a terminal replacing your Earthdata login username and password:
echo "machine urs.earthdata.nasa.gov login EARTHDATA_LOGIN password EARTHDATA_PASSWORD" > ~/.netrc
chmod 0600 .netrc
Note that the script below should prompt you with your Earthdata Login username and password if a .netrc file does not exist.
# os.chmod('/home/jovyan/.netrc', 0o600) #only necessary on snowex hackweek jupyterhub
%run './scripts/nsidc-download_SNEX20_BSU_CMP_SWE.001'
print('SnowEx20 Grand Mesa IOP BSU 1 GHz Multi-polarization GPR CMP Snow Water Equivalent download complete')
Querying for data:
https://cmr.earthdata.nasa.gov/search/granules.json?provider=NSIDC_ECS&sort_key[]=start_date&sort_key[]=producer_granule_id&scroll=true&page_size=2000&short_name=SNEX20_BSU_CMP_SWE&version=001&version=01&version=1&temporal[]=2020-01-31T00:00:00Z,2020-02-01T23:59:59Z
Found 2 matches.
Downloading 14 files...
01/14: SNEX20_BSU_CMP_SWE_01312020_CMP1_HH.csv
[============================================================] 100% 576.9kB/s
02/14: SNEX20_BSU_CMP_SWE_01312020_CMP1_HH.png
[============================================================] 100% 855.3kB/s
03/14: SNEX20_BSU_CMP_SWE_01312020_CMP2_HH.csv
[============================================================] 100% 749.5kB/s
04/14: SNEX20_BSU_CMP_SWE_01312020_CMP2_HH.png
[============================================================] 100% 892.5kB/s
05/14: SNEX20_BSU_CMP_SWE_01312020_CMP3_HH.csv
[============================================================] 100% 622.9kB/s
06/14: SNEX20_BSU_CMP_SWE_01312020_CMP3_HH.png
[============================================================] 100% 859.9kB/s
07/14: SNEX20_BSU_CMP_SWE_01312020.xml
[============================================================] 100% 4.6MB/s
08/14: SNEX20_BSU_CMP_SWE_02012020_CMP1_HH.csv
[============================================================] 100% 617.2kB/s
09/14: SNEX20_BSU_CMP_SWE_02012020_CMP1_HH.png
[============================================================] 100% 939.4kB/s
10/14: SNEX20_BSU_CMP_SWE_02012020_CMP2_HH.csv
[============================================================] 100% 562.2kB/s
11/14: SNEX20_BSU_CMP_SWE_02012020_CMP2_HH.png
[============================================================] 100% 797.9kB/s
12/14: SNEX20_BSU_CMP_SWE_02012020_CMP3_HH.csv
[============================================================] 100% 830.4kB/s
13/14: SNEX20_BSU_CMP_SWE_02012020_CMP3_HH.png
[============================================================] 100% 878.5kB/s
14/14: SNEX20_BSU_CMP_SWE_02012020.xml
[============================================================] 100% 4.5MB/s
SnowEx20 Grand Mesa IOP BSU 1 GHz Multi-polarization GPR CMP Snow Water Equivalent download complete
Option 2: Additional data access services: API Access¶
Data can be accessed directly from our HTTPS file system as described in the aforementioned tutorial, or through the NSIDC DAAC’s Application Programming Interface (API).
What is an API? You can think of an API as a middle man between an application or end-use (in this case, us) and a data provider. Here, the data provider is both the Common Metadata Repository (CMR) housing data information, and NSIDC as the data distributor. These APIs are generally structured as a URL with a base plus individual key-value-pairs separated by ‘&’. This option is beneficial for those of you who want to incorporate data access directly into your visualization and analysis coding workflow, without the need to utilize the NSIDC website. This method is also reproducible and documented to ensure data provenance.
This API offers you the ability to order data using specific temporal and spatial filters. These options can be requested in a single access command without the need to script against our data directory structure. See the programmatic access guide for more information on API options.
Create the data request API endpoint¶
Programmatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above.
The cell below sets up the API request URL using our data search parameters as well as a few other configuration parameters. We will first create a string of key-value-pairs from our data dictionary and we’ll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below.
#Set NSIDC data access base URL
base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'
#Set the request mode to asynchronous, "no" processing agent (no subsetting or reformatting services available), and optionally removing metadata delivery
param_dict['request_mode'] = 'async'
param_dict['agent'] = 'NO'
param_dict['INCLUDE_META'] ='N' #optional if you do not wish to receive the associated metadata files with each science file.
param_string = '&'.join("{!s}={!r}".format(k,v) for (k,v) in param_dict.items()) # Convert param_dict to string
param_string = param_string.replace("'","") # Remove quotes
api_list = [f'{base_url}?{param_string}']
api_request = api_list[0]
print(api_request) # Print API base URL + request parameters
https://n5eil02u.ecs.nsidc.org/egi/request?short_name=SNEX20_BSU_CMP_SWE&version=1&polygon=-108.2352445938561,38.98556907427165,-107.85284607930835,38.978765032966244,-107.85494925720668,39.10596902171742,-108.22772795408136,39.11294532581687,-108.2352445938561,38.98556907427165&temporal=2020-01-01T00:00:00Z,2020-12-31T23:59:59Z&request_mode=async&agent=NO&INCLUDE_META=N
Input Earthdata Login credentials¶
For our API access option, An Earthdata Login account is required to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register.
# Start authenticated session with Earthdata Login to allow for data downloads:
def setup_earthdata_login_auth(endpoint: str='urs.earthdata.nasa.gov'):
netrc_name = "_netrc" if system()=="Windows" else ".netrc"
try:
username, _, password = netrc.netrc(file=join(expanduser('~'), netrc_name)).authenticators(endpoint)
except (FileNotFoundError, TypeError):
print('Please provide your Earthdata Login credentials for access.')
print('Your info will only be passed to %s and will not be exposed in Jupyter.' % (endpoint))
username = input('Username: ')
password = getpass('Password: ')
manager = request.HTTPPasswordMgrWithDefaultRealm()
manager.add_password(None, endpoint, username, password)
auth = request.HTTPBasicAuthHandler(manager)
jar = CookieJar()
processor = request.HTTPCookieProcessor(jar)
opener = request.build_opener(auth, processor)
request.install_opener(opener)
setup_earthdata_login_auth(endpoint="urs.earthdata.nasa.gov")
Download data¶
Download data using the requests library. The data will be downloaded directly to this directory in a new Outputs folder. The progress of each order will be reported.
def request_nsidc_data(API_request):
"""
Performs a data customization and access request from NSIDC's API/
Creates an output folder in the working directory if one does not already exist.
:API_request: NSIDC API endpoint; see https://nsidc.org/support/how/how-do-i-programmatically-request-data-services for more info
on how to configure the API request.
"""
path = str(os.getcwd() + '/Outputs') # Create an output folder if the folder does not already exist.
if not os.path.exists(path):
os.mkdir(path)
base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'
r = request.urlopen(API_request)
esir_root = ET.fromstring(r.read())
orderlist = [] # Look up order ID
for order in esir_root.findall("./order/"):
orderlist.append(order.text)
orderID = orderlist[0]
statusURL = base_url + '/' + orderID # Create status URL
print('Order status URL: ', statusURL)
request_response = request.urlopen(statusURL) # Find order status
request_root = ET.fromstring(request_response.read())
statuslist = []
for status in request_root.findall("./requestStatus/"):
statuslist.append(status.text)
status = statuslist[0]
while status == 'pending' or status == 'processing': #Continue loop while request is still processing
print('Job status is ', status,'. Trying again.')
time.sleep(10)
loop_response = request.urlopen(statusURL)
loop_root = ET.fromstring(loop_response.read())
statuslist = [] #find status
for status in loop_root.findall("./requestStatus/"):
statuslist.append(status.text)
status = statuslist[0]
if status == 'pending' or status == 'processing':
continue
if status == 'complete_with_errors' or status == 'failed': # Provide complete_with_errors error message:
messagelist = []
for message in loop_root.findall("./processInfo/"):
messagelist.append(message.text)
print('Job status is ', status)
print('error messages:')
pprint(messagelist)
if status == 'complete' or status == 'complete_with_errors':# Download zipped order if status is complete or complete_with_errors
downloadURL = 'https://n5eil02u.ecs.nsidc.org/esir/' + orderID + '.zip'
print('Job status is ', status)
print('Zip download URL: ', downloadURL)
print('Beginning download of zipped output...')
zip_response = request.urlopen(downloadURL)
with zipfile.ZipFile(io.BytesIO(zip_response.read())) as z:
z.extractall(path)
print('Download is complete.')
else: print('Request failed.')
# Clean up Outputs folder by removing individual granule folders
for root, dirs, files in os.walk(path, topdown=False):
for file in files:
try:
shutil.move(os.path.join(root, file), path)
except OSError:
pass
for name in dirs:
os.rmdir(os.path.join(root, name))
return
# NOTE: downloads ~ 200MB of CSV files
request_nsidc_data(api_request)
Order status URL: https://n5eil02u.ecs.nsidc.org/egi/request/5000003841849
Job status is complete
Zip download URL: https://n5eil02u.ecs.nsidc.org/esir/5000003841849.zip
Beginning download of zipped output...
Download is complete.
Read in SnowEx data¶
This SnowEx data set is provided in CSV. A Pandas DataFrame is used to easily read in data.
snowex_path = './SNEX20_BSU_CMP_SWE_01312020_CMP1_HH.csv' # Define local filepath
df = pd.read_csv(snowex_path, sep=',')
df.head()
| UTCyear | UTCdoy | UTCtod | UTMzone | Easting | Northing | Elevation | t0LMO1 | vLMO1 | zLMO1 | rhoLMO1 | sweLMO1 | t0NMO1 | vNMO1 | zNMO1 | rhoNMO1 | sweNMO1 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.289799 | 0.207728 | 20.772782 | 524.565927 | 108.966936 | 8.439704 | 0.223405 | 94.273613 | 404.704691 | 381.529734 |
| 1 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.309042 | 0.207476 | 20.747610 | 526.638240 | 109.264847 | 8.325702 | 0.221085 | 92.034483 | 421.369388 | 387.805137 |
| 2 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.319745 | 0.207463 | 20.746335 | 526.743281 | 109.279928 | 8.313888 | 0.222871 | 92.646168 | 408.511172 | 378.469946 |
| 3 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.270248 | 0.207216 | 20.721626 | 528.782649 | 109.572363 | 8.322683 | 0.216686 | 90.170317 | 453.953920 | 409.331691 |
| 4 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.094891 | 0.201603 | 20.160328 | 576.455359 | 116.215292 | 8.571260 | 0.219020 | 93.863736 | 436.504376 | 409.719313 |
Extract date values¶
The collection date needs to be extracted from the UTCyear and UTCdoyparameters:
df['date'] = pd.to_datetime(df['UTCyear'] * 1000 + df['UTCdoy'], format="%Y%j")
df.reset_index(drop=True, inplace=True)
df.head()
| UTCyear | UTCdoy | UTCtod | UTMzone | Easting | Northing | Elevation | t0LMO1 | vLMO1 | zLMO1 | rhoLMO1 | sweLMO1 | t0NMO1 | vNMO1 | zNMO1 | rhoNMO1 | sweNMO1 | date | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.289799 | 0.207728 | 20.772782 | 524.565927 | 108.966936 | 8.439704 | 0.223405 | 94.273613 | 404.704691 | 381.529734 | 2020-01-31 |
| 1 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.309042 | 0.207476 | 20.747610 | 526.638240 | 109.264847 | 8.325702 | 0.221085 | 92.034483 | 421.369388 | 387.805137 | 2020-01-31 |
| 2 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.319745 | 0.207463 | 20.746335 | 526.743281 | 109.279928 | 8.313888 | 0.222871 | 92.646168 | 408.511172 | 378.469946 | 2020-01-31 |
| 3 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.270248 | 0.207216 | 20.721626 | 528.782649 | 109.572363 | 8.322683 | 0.216686 | 90.170317 | 453.953920 | 409.331691 | 2020-01-31 |
| 4 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.094891 | 0.201603 | 20.160328 | 576.455359 | 116.215292 | 8.571260 | 0.219020 | 93.863736 | 436.504376 | 409.719313 | 2020-01-31 |
Convert to Geopandas dataframe to provide point geometry¶
According to the SnowEx documentation, the data are available in UTM Zone 12N so we’ll set to this projection to allow for geospatial analysis:
gdf_utm= gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.Easting, df.Northing), crs='EPSG:32612')
gdf_utm.head()
| UTCyear | UTCdoy | UTCtod | UTMzone | Easting | Northing | Elevation | t0LMO1 | vLMO1 | zLMO1 | rhoLMO1 | sweLMO1 | t0NMO1 | vNMO1 | zNMO1 | rhoNMO1 | sweNMO1 | date | geometry | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.289799 | 0.207728 | 20.772782 | 524.565927 | 108.966936 | 8.439704 | 0.223405 | 94.273613 | 404.704691 | 381.529734 | 2020-01-31 | POINT (742509.199 4324359.885) |
| 1 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.309042 | 0.207476 | 20.747610 | 526.638240 | 109.264847 | 8.325702 | 0.221085 | 92.034483 | 421.369388 | 387.805137 | 2020-01-31 | POINT (742509.199 4324359.885) |
| 2 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.319745 | 0.207463 | 20.746335 | 526.743281 | 109.279928 | 8.313888 | 0.222871 | 92.646168 | 408.511172 | 378.469946 | 2020-01-31 | POINT (742509.199 4324359.885) |
| 3 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.270248 | 0.207216 | 20.721626 | 528.782649 | 109.572363 | 8.322683 | 0.216686 | 90.170317 | 453.953920 | 409.331691 | 2020-01-31 | POINT (742509.199 4324359.885) |
| 4 | 2020 | 31 | 181632.874894 | 12S | 742509.198829 | 4.324360e+06 | 3048.586525 | 0.094891 | 0.201603 | 20.160328 | 576.455359 | 116.215292 | 8.571260 | 0.219020 | 93.863736 | 436.504376 | 409.719313 | 2020-01-31 | POINT (742509.199 4324359.885) |
Additional data imagery services¶
NASA Worldview and the Global Browse Imagery Service¶
NASA’s EOSDIS Worldview mapping application provides the capability to interactively browse over 900 global, full-resolution satellite imagery layers and then download the underlying data. Many of the available imagery layers are updated within three hours of observation, essentially showing the entire Earth as it looks “right now.”
Several MODIS snow data products from NSIDC include high-resolution browse imagery available through NASA Worldview, including “MODIS/Terra Snow Cover Daily L3 Global 500m SIN Grid, Version 61”. This layer can be downloaded as various image files including GeoTIFF using the snapshot feature at the top right of the page. This link presents the MOD10A1 NDSI layer over our time and area of interest: https://go.nasa.gov/35CgYMd.
Additionally, the NASA Global Browse Imagery Service provides up to date, full resolution imagery for select NSIDC DAAC data sets as web services including WMTS, WMS, KML, and more. These layers can be accessed in GIS applications following guidance on the GIBS documentation pages.