The main objective of the RepreSent project is to capitalize on the potential of artificial intelligence (AI) and Earth observation (EO) by exploiting the non-supervised learning paradigms. In this context, it is essential to come up with non-supervised learning-based solutions for impactful use cases that use unlabeled EO data.
The project started on April 1st, 2022 with the project kick-off on April 6th, 2022. The project reached successfully the milestone MS-1 in July 11th, 2022 and the milestone MS-2 in October 12th, 2022. The final review of the project took place successfully on May 11th, 2023 at ESRIN.
Objectives
Firstly, the consortium will investigate the non-supervised learning-based methods for EO, as one of project technical objective. Towards harnessing on the effectiveness of non-supervised learning and exploiting the multitude of unlabeled data, successful feature extractors will be built based on self-supervised learning-based pre-training and transfer learning from pre-trained networks.
These techniques are grouped into three main categories:
- Self-supervised learning for uni-temporal tasks (e.g., Deep Clustering, Contrastive Learning, Bootstrap Your Own Latent, Meta-Learning),
- Change detection (e.g., Deep change vector analysis, Deep multi-temporal segmentation, Temporal contrastive learning),
- Time-series anomaly detection (e.g., LSTM based unsupervised approach, Graph neural networks based on a self‑supervised approach).
DLR and EPFL will use their AI4EO network and they will establish contacts with other research groups addressing similar or related thematic areas in the AI/EO/NS community.
Secondly, the validation of the technical objectives is fulfilled by the consortium by defining five use cases. These use cases are focus on the challenges of EO where either labelled data are scarce or the successful application of supervised methods requires many labels that are tedious or costly to collect. The validation was done on five use cases related to:
- UC1. Forest disturbance monitoring
- UC2. Automated Land Cover mapping
- UC3. Anomaly detection in long time series of PS-P InSAR
- UC4. Cloud detection, and removal
- UC5. Forest biomass estimation
For these use cases, the required EO data is exploit the openly available Sentinel-1, Sentinel-2, and Landsat archives.
The RepreSent EO use cases are tackling core business and scientific questions since they are tightly connected to e-GEOS and VTT core business areas and active projects (InSAR/ground motion and forestry, respectively). This is also ensuring a close link to users and stakeholders in the different sectors where the two partners are very well known.
Finally, the non-supervised learning methods developed by the consortium were evaluated based on usual quantitative performance metrics along with qualitative analyses commenting on their generalization capability and versatility on different EO cases.
AI4EOcommunity of players
Each use case proposed within the project had a user target group. For example, for the use case on forestry, VTT used a wide community of academic and forestry users including forest owners via the Forestry TEP. On May 4th, 2023, an online workshop on Self-supervised learning (SSL) in Earth Observation based forest inventory was conducted by VTT and the ESA RepreSent project consortium.
For the use case on anomaly detection, e-GEOS is in contact with several customers interested in the detection of anomalous points.
The project partners were involved in the organization of special sessions at different conferences related to the topic of the RepreSent project. In June 2022 it was organized by EPFL and DLR the first special session at Living Planet Symposium (LPS’22) in Bonn, Germany entitled “Representation learning in remote sensing: from unsupervised to self- and meta-learning”, while in July 2022 an invited session was organized by EPFL and VTT at the ISPRS congress 2022 in Nice, France entitled “Unsupervised and weakly supervised deep learning for EO”. For 2023, the RepreSent team has organised a community invited session at IGARSS 2023 in Pasadena, USA. The session is titled “Representation learning in remote sensing” and will be composed of two sub sections (out of which of the accepted papers, two belong to the members of the consortium). Members of the team will chair the sessions in Pasadena. EPFL together with ESA are co-organiser of the EarthVision workshop at CVPR in 2022 and 2023.
Finally, during the final review of the project at ESA in ESRIN, two public sessions were organised by the RepreSent consortium together with ESA under the topic AI4EO on SSL.
Extension of RepreSent for Scaling-up
The initial phase of the RepreSent project enabled us to delve into the field of representation learning, showcasing its vast potential in making EO data and methodologies broadly accessible without requiring extensive effort. From a technical perspective, the project has been successful, as evidenced by the positive feedback received from the academic community towards our publications and presentations, and the increasing interest in exploring, replicating, and expanding our developed methodologies. Furthermore, achieving our current TRL has initiated communication with a variety of stakeholders, including those in fields such as forest farming, environmental monitoring, and urban planning.
The project extension started on February 1st, 2024 with the project kick-off on February 7th, 2024.The main objectives are
- Our first objective focuses on improvement of accuracy and timeliness of EO based forest mapping using multi-temporal and multi-sensor data (in contrast to studied earlier bi-temporal and single/bi-sensor approaches), leveraging self-supervised learning (SSL) methods
- Our second objective is to enhance our ability to detect building anomalies. Given the varied and changing nature of urban environments, we aim to expand the area of study. This expansion will allow us to better understand the patterns of anomalies across different urban landscapes.
- Our third objective focuses on the refinement of cloud detection methods. Recognizing the potential of self-supervised learning in cloud detection, achieving comparable accuracy as state-of-the-art supervised methods on a small-scale cloud dataset, we plan to extend our experiments to the CloudSEN12 dataset. This will provide us with a global scale of data for validating our approach
- Our final objective pertains to the continued and enhanced dissemination of our project’s outcomes. We aim to reach a broader scientific audience by publishing our findings in respected peer-reviewed journals.
The extension consortium (DLR, VTT, and e-GEOS) considers three use cases for which a number of users have shown their interest. These use cases are:
- UC1. Multi-temporal and multi-sensor forest mapping
- UC2. Building anomaly detection
- UC3. Cloud detection
The RepreSent CCN project extension passed the middle term review (MTR on July 18th 2024) period defined in the contract and all documents have been successfully validated by ESA. The consortium team members are preparing the submission of different contributions, related to each use case, promoting the project results at different conferences and journals. The final presentation of the project extension is planned for December 2024.
Conference papers
- C.O. Dumitru, R. Kuzu, L. Bagaglini, F. Santarelli, Building Anomaly Detection with Self-supervised Learning. Case Study: The City of Bucharest, Romania, URBIS24 Workshop, Frascati, 16-18 September 2024 (poster).
- R. Kuzu, Y. Wang, O. Dumitru, L. Bagaglini, G. Pasquali, F. Santarelli, F. Trillo, S. Saha, and X. Zhu, An Unsupervised Anomaly Detection Problem in Urban InSAR-PSP Long Time-series, EGU 2023, Vienna, 23-28 April 2023.
- O. Antropov, M. Molinier, R. Kuzu, L. Hughes, M. Rußwurm, D. Tuia, O. Dumitru, S. Ge, S. Saha, X. Zhu, Semi-Supervised Deep Learning Representations in Earthe Observation Based Forest Management, IGRASS 2023, Pasadena, 16-21 July 2023.
- M. Rußwurm, L. Hughes, G. Pasquali, O. Dumitru, D. Tuia, Detection of Settlements in Tanzania and Mozambique by Many Regional Few-Shot Models, IGRASS 2023, Pasadena, 16-21 July 2023.
- C.O. Dumitru, R. Kuzu, L. Hughes, M. Russwurm, D. Tuia, O. Antropov, M. Molinier, G. Pasquali, L. Bagaglini, A. Rösel, and X.X. Zhu, RepreSent: Non-supervised Representation Learning for Sentinels, Neue Perspektiven der Erdbeobachtung Symposium, Bonn, Germany, 26-28 June 2023 (poster).
Additional achievements:
