TerraCatalogue: Product Download
Finding product links
The responses from both the OpenSearch API and the STAC API can be used for downloading the products in the TerraCatalogue. Authentication is required for the product download.
In OpenSearch, the product level search response includes the following links for most of our EO products:
- previews provide links to quicklooks.
- alternate provides a link to INSPIRE product metadata.
- related provides links to auxiliary data (e.g. sceneclassification, RAA, AOT, …)
- data provides links to the different files (bands) of the product.
The standard OpenSearch responses include HTTP download links.
In STAC, the purpose of each asset can be derived from the ‘roles’ assigned to it. The following roles are often present in the different assets of a STAC Item:
- thumbnail or overview provides a link to the preview image of the product.
- metadata provides a link to an additional product metadata file (eg. INSPIRE).
- data provide links to the actual data assets (e.g. RED, BLUE, B08, FAPAR, ….) of the product.
The default .href of each asset is the HTTP download link. Additionally, the local file path on the Terrascope platform is provided in the .alternate.local.href field.
It is not necessary to download the products on the Terrascope platform. The data is locally available on the user VM’s and Notebooks.
The responses from the OpenSearch API can be modified to return direct access links to the local files in your Terrascope environment. This makes local data discovery easier, and allows for easy transfer of processing logic between your own environment and the Terrascope Platform. To request direct access links, add the accessedFrom parameter with the value MEP to your OpenSearch API request.
This alters the href value of each data link to point to a local file instead of a download link, as can be seen in the example below. This will then allow you to use the local path in your data processing logic.
{
"length":405612,
"type":"image/tiff",
"title":"TOC-B01_60M",
"href":"file:///data/MTDA/TERRASCOPE_Sentinel2/TOC_V2/2020/01/01/S2A_20200101T073311_36KWC_TOC_V200/S2A_20200101T073311_36KWC_TOC-B01_60M_V200.tif"
}Authentication
Once you’ve found the neccessary products you need to authenticate your requests to download the files. Terrascope supports several authentication methods, which can be useful in different circumstances.
Interactive authentication
This is the easiest method and can be handy when you need to download individual files directly. When you visit a download link in your browser to download a file manually you will be prompted to authenticate your download. Terrascope supports multiple ways of authentication:
- Via the Terrascope platform. Read more about creating a Terrascope account here.
- Via an EGI account.
Once you’ve followed the steps of the login flow the file download should begin.
STAC tools
You can use the tools from the STAC ecosystem to discover and download data. For downloading, we recommend the stac-asset library. This library has support for authorizing HTTP downloads with OAuth2 access tokens. We provide an example notebook on how to interact with the STAC API and the download service using these tools on GitHub.
Terracatalogueclient
VITO provides a Python package to facilitate programmatic interaction with the TerraCatalogue. The package includes several convenience functions to reduce the complexity of authentication. Please consult the documentation of the package for usage examples, or explore a sample jupyter notebook which walks you through the entire flow. This notebook is available on GitHub or in the Terrascope jupyterhub environment
Machine authentication
If you do not use Python or want to build your own processing logic it is also possbile to authenticate by setting the Authorization header of your HTTP requests. Terrascope supports both the Basic and Bearer credentials.
An example notebook has been made available which walks you through the usage of both authentication methods. It can be viewed on GitHub or directly in the Terrascope JupyterLab envrionment.
Basic auth
Basic authentication uses a Base64 encoded combination of your Terrascope username and password to complete the Authorization header of your request. An example of a basic authentication header can be found here.
Bearer auth
Authenticating using bearer tokens requires an access token. This token is requested using your terrascope credentials and is valid for a limited time (300 seconds). When the token expires you can either request a new token using your credentials or by using the refresh token which was included in the initial token request response. Please consult the documentation for additional information on token-based authentication.
When you need to download many products it can be useful to handle token refreshes automatically. Most languages offer ways to handle this process. An example implementation has been made in python and can be found in the sample notebook linked above.
Rate limiting
Downloads are rate limited based on IP address:
- 80 requests per 10 seconds
- 200 requests per 60 seconds
GDAL support
With basic authentication, the products can also be accessed via the GDAL network based file system. For example reading a WorldCover product:
gdalinfo /vsicurl/https://${USERNAME}:${PASSWORD}@services.terrascope.be/download/WORLDCOVER/ESA_WORLDCOVER_10M_2020_V100/MAP/ESA_WorldCover_10m_2020_v100_N00E006_Map/ESA_WorldCover_10m_2020_v100_N00E006_Map.tif