TerraCatalogue: Product Download
Finding product links
The responses from both the OpenSearch API and the STAC API can be used for downloading the products in the TerraCatalogue. Authentication is required for the product download.
In OpenSearch, the product level search response includes the following links for most of our EO products:
- previews provide links to quicklooks.
- alternate provides a link to INSPIRE product metadata.
- related provides links to auxiliary data (e.g. sceneclassification, RAA, AOT, …)
- data provides links to the different files (bands) of the product.
In STAC, the following assets are often included in the item search:
- QUICKLOOK provide links to quicklooks.
- WMS provides a link to the WMS URL of the product.
- Inspire metadata provides a link to product metadata.
- BANDNAME provide links to the different BANDS (e.g. SCENECLASSIFICATION, RED, BLEU, B08, FAPAR, ….) of the product.
The standard OpenSearch and STAC responses include HTTP download links.
It is not necessary to download the products on the Terrascope platform. The data is locally available on the user VM’s and Notebooks.
The responses from the OpenSearch API can be modified to return direct access links to the local files in your Terrascope environment. This makes local data discovery easier, and allows for easy transfer of processing logic between your own environment and the Terrascope Platform. To request direct access links, add the accessedFrom parameter with the value MEP to your OpenSearch API request.
This alters the href value of each data link to point to a local file instead of a download link, as can be seen in the example below. This will then allow you to use the local path in your data processing logic.
{
"length":405612,
"type":"image/tiff",
"title":"TOC-B01_60M",
"href":"file:///data/MTDA/TERRASCOPE_Sentinel2/TOC_V2/2020/01/01/S2A_20200101T073311_36KWC_TOC_V200/S2A_20200101T073311_36KWC_TOC-B01_60M_V200.tif"
}This functionality is only available in OpenSearch. Work is ongoing to include it in the STAC API, and the documentation will be updated when it is also available there.
Authentication
Once you’ve found the neccessary products you need to authenticate your requests to download the files. Terrascope supports several authentication methods, which can be useful in different circumstances.
Interactive authentication
This is the easiest method and can be handy when you need to download individual files directly. When you visit a download link in your browser to download a file manually you will be prompted to authenticate your download. Terrascope supports multiple ways of authentication:
- Via the Terrascope platform. Read more about creating a Terrascope account here.
- Via an EGI account.
Once you’ve followed the steps of the login flow the file download should begin.
Terracatalogueclient
VITO provides a python package to facilitate programmatic interaction with the TerraCatalogue. The package includes several convenience functions to reduce the complexity of authentication. Please consult the documentation of the package for usage examples, or explore a sample jupyter notebook which walks you through the entire flow. This notebook is available on GitHub or in the Terrascope jupyterhub environment
Machine authentication
If you do not use Python or want to build your own processing logic it is also possbile to authenticate by setting the Authorization header of your HTTP requests. Terrascope supports both the Basic and Bearer credentials.
An example notebook has been made available which walks you through the usage of both authentication methods. It can be viewed on GitHub or directly in the Terrascope JupyterLab envrionment.
Basic auth
Basic authentication uses a Base64 encoded combination of your Terrascope username and password to complete the Authorization header of your request. An example of a basic authentication header can be found here.
Bearer auth
Authenticating using bearer tokens requires an access token. This token is requested using your terrascope credentials and is valid for a limited time (300 seconds). When the token expires you can either request a new token using your credentials or by using the refresh token which was included in the initial token request response. Please consult the documentation for additional information on token-based authentication.
When you need to download many products it can be useful to handle token refreshes automatically. Most languages offer ways to handle this process. An example implementation has been made in python and can be found in the sample notebook linked above.
Rate limiting
Downloads are rate limited based on IP address:
- 80 requests per 10 seconds
- 200 requests per 60 seconds
GDAL support
With basic authentication, the products can also be accessed via the GDAL network based file system. For example reading a WorldCover product:
gdalinfo /vsicurl/https://${USERNAME}:${PASSWORD}@services.terrascope.be/download/WORLDCOVER/ESA_WORLDCOVER_10M_2020_V100/MAP/ESA_WorldCover_10m_2020_v100_N00E006_Map/ESA_WorldCover_10m_2020_v100_N00E006_Map.tif