It’s a great time to work in Earth Observation, with an unprecedented number of missions acquiring high-quality, freely accessible data. These are complemented by a whole ecosystem of analytic tools and software.
The ESA Earth Observation Program (EOP) supports EO downstream industry in developing novel applications, and numerous innovative results are generated every year in various thematic domains. However, ensuring that such a wealth of results remains easily accessible to the larger EO community after the end of project funding is not straightforward.
The sustainability challenge in EO applications
EO project results range from key value-added data products to algorithms and workflows but might also include reports, presentations or the knowledge contained in a user forum. Results such as algorithms and workflows quickly become obsolete due to software dependencies, lack of code maintenance, or evolving input data product specifications. On the other hand, value-added products that were generated in EOP projects quickly become inaccessible after the related project ends. Ensuring that EO results such as algorithms/workflows or value-added products remain Findable, Accessible, Interoperable and Reusable (FAIR) is a key challenge which is also referred to as the FAIR data principles.
Another shortcoming encountered in the EOP projects is that while projects typically require similar resources to develop and promote their results (e.g., web portal, user forum, webGIS), they rarely make use of existing solutions. This constant reinventing the wheel limits ESA’s ability to sustain these project resources on the longer term, while also diverting resources from core objectives such as advancing an innovative application.
Fortunately, today many technologies can help working more effectively and the EO platform ecosystem is gradually converging with the uptake and adoption of these technologies. Nevertheless, there is a strong need for initiatives to streamline relevant processes and effectively federate resources in the fragmented European EO platform landscape. This is where the Application Propagation Environments (APEx) comes in.
APEx: Application Propagation Environments, here to bring things together
“APEx seeks to improve the availability of EO R&D results for the EO community, by fostering the adoption of FAIR data principles and streamlining the transition of application algorithms into operational cloud service implementations that are available for the community to reuse and build upon.”
Kicked-off in April 2024, due to run for 2+3 years initially, APEx does not intend to create a new EO platform but aggregates and integrates existing resources and services across the EO platform ecosystem. Its main goals are to address:
- Last km challenge: systematic availability of application algorithms as ready-to-use (and to reuse by the community) Information-as-a-Service elements;
- Production readiness: performance-optimised algorithms, deployable and scalable on different cloud environments;
- Global scaling: simplified large-scale geographic upscaling (continental-global) by abstracting all data management related considerations from the user;
- Cloudification: availability of EO toolboxes functionality as ready-to-use services in the cloud-based platform ecosystem;
- Sustainability: long-term and open accessibility of value-added products, workflows/algorithms, documentation, etc.;
- Integrated project environments: on-demand deployment of configurable and integrated cloud-based project environments;
- Open federation: promote interoperability across European and national infrastructures supporting use of APEx services, algorithms and products.
The APEx portal is the main entry point to services and solutions and is accessible at: https://apex.esa.int/. Here follows an outline of the main solutions and what they encompass:
APEx and interoperability
With the solutions and approach described above, APEx seeks to build an open federation that effectively links resources and services from the European EO platform ecosystem. This is done by incentivising interoperability and building on synergies and complementarity with two other key initiatives, the EOEPCA+ Common Architecture project and the ESA Network of Resources. Within the collaboration with EOEPCA+ APEx contributes to the requirements and use cases for the different open source building blocks which will then also be adopted and integrated into the APEx services. The linkages with the ESA NOR include offering dedicated APEx solutions and publishing ready-to-use Information-as-a-Service offers in the NOR portfolio.
In additional, APEx maintains and evolves a set of Interoperability and Compliance Guidelines which define what algorithm developers and EO platforms should comply with in order to expose and integrate APEx aggregated solutions. Key elements of these guidelines include for example the use of cloud-optimised raster formats or the use of STAC encoded metadata. Altogether, these guidelines and strategic collaborations will help to further consolidate the European EO Platform ecosystem towards FAIR compliance.
APEx in action: success stories
Less than one year after APEx launch, the exploitation of its services has been already successfully demonstrated in a few use cases.
On-demand wind turbine detection: operationalisation with algorithm porting and packaging
The benefits of cloud-based implementation are nicely demonstrated by the GTIF Wind turbine detection algorithm. Originally developed as part of the Green Transition Information Factory (GTIF) Demonstrator for Austria, DHI GRAS (DK) developed a machine-learning algorithm that detects wind turbines from Sentinel-2 imagery. The GTIF explorer provided detection results based on one year’s worth of data, allowing users to overlay them with Sentinel-2 imagery for analysis.
Soon after this went live, demand quickly grew. Stakeholders expressed interest in expanding the detection at national scale for recent years and extending it to other countries. To meet these requirements, DHI ported the original python algorithm to an openEO-based implementation, following the APEx interoperability guidelines, unlocking several benefits for example:
- Abstraction of data input/output considerations: the algorithm can be easily applied to different geographic areas and input/output data are automatically set-up. Updates to the Sentinel-2 L2A product specification are automatically resolved.
- Scalability: taking advantage of dynamic allocation of compute resources in the openEO backends makes running the algorithm over larger areas more efficient.
- Interoperability: the resulting openEO service (a so-called User Defined Process) becomes available on any EO platform that supports the openEO API and that hosts the S2 L2A collection.
Moreover, DHI now offers this algorithm service at a commercial level in the ESA Network of Resources (NOR), via openEO platform, as the algorithm hosting environment. Users can commercially consume the service or apply for sponsorship, creating a revenue stream for both DHI and the platform operator.
This case study highlights how an EO algorithm can be successfully operationalised, transitioning from research to a scalable, commercial cloud service. APEx algorithms services seek to streamline this process by providing tailored support and, soon, offering tools to support project teams with the technical implementation.
Global Sentinel-2 burned area mapping: huge cost improvements with algorithm optimisation
The ESA Climate Change Initiative (CCI) program develops Essential Climate Variables (ECVs) that evidence our changing climate and inform international action. One product of the Fire ECV is burned area which – thanks to the Fire CCI project – is now derived from Sentinel-2 rather than coarse resolution data sources.
After the fire CCI burned area algorithm reached scientific maturity, the next challenge was scaling it to a global level. Initial cost assessments for cloud computing resources showed that US-based public clouds (AWS and Google) were cheaper than European options such as CreoDIAS. However, the estimated expense for global production ranged between 200,000€ and 300,000€ – a significant investment.
To reduce costs, an alternative strategy was adopted: a specialised company conducted an in-depth analysis of the source code, identifying key areas for performance optimisation. After implementing these critical improvements, processing efficiency was drastically enhanced, reducing the total cost of global production to just 75,000€.
The optimised algorithm was then integrated as a data processing as a service offering within the ESA Network of Resources (NoR), making it more accessible and cost-effective for users.
This example illustrates the value of source code optimisation in improving computational efficiency, especially since much of the EO R&D code is not originally designed for large-scale production. The APEx algorithm services will streamline this process and build an ecosystem of FAIR-complaint EO services that the EO community can reuse and build upon.
How to work with APEx?
The APEx portal is the main entry point to solutions and services. Key APEx resources, such as the algorithm services and data catalogues, are openly accessible. However, users must secure access rights to the hosting environment where the algorithms are deployed. ESA Network of Resources can be considered to obtain sponsorship and license packages.
The APEx algorithm services comprise valuable resources on how to transform applications and algorithms into openEO processes or application packages. Tailored support for algorithm porting, algorithm hosting or algorithm optimisation, is currently only offered on a case-by-case basis and preference is currently given to ESA EO projects. Future plans include expanding services to a broader community and making them available in the ESA NOR (i.e. also for non-ESA activities). The same applies to APEx project environments.
For more information and collaboration opportunities, please visit the APEx portal and get in touch via the indicated communication channels.