image: Caption: Illustration of the data used for training the tiny cloud classification model (left), and the predictions on new scenes (right). The entire training process took about 1.5 seconds, including the time for encoding the entire training dataset, and 10 epochs of training a classification model. Image credit: Sentinel-2 data (ESA) processed by Vít Růžička view more
Credit: Caption: Illustration of the data used for training the tiny cloud classification model (left), and the predictions on new scenes (right). The entire training process took about 1.5 seconds, including the time for encoding the entire training dataset, and 10 epochs of training a classification model. Image credit: Sentinel-2 data (ESA) processed by Vít Růžička
- Researchers used an innovative machine learning approach to develop a tiny model capable of running on a satellite’s limited processing power;
- The trained model successfully detected cloud cover in satellite images in around a tenth of a second;
- The model could easily be adapted to enable automated decision making for a range of purposes, from disaster management to deforestation.
For the first time, a project led by the University of Oxford has trained a machine learning model in outer space, on board a satellite. This achievement could revolutionise the capabilities of remote-sensing satellites by enabling real-time monitoring and decision making for a range of applications.
Data collected by remote-sensing satellites is fundamental for many key activities, including aerial mapping, weather prediction, and monitoring deforestation. Currently, most satellites can only passively collect data, since they are not equipped to make decisions or detect changes. Instead, data has to be relayed to Earth to be processed, which typically takes several hours or even days. This limits the ability to identify and respond to rapidly emerging events, such as a natural disaster.
To overcome these restrictions, a group of researchers led by DPhil student Vít Růžička (Department of Computer Science, University of Oxford), took on the challenge of training the first machine learning program in outer space. During 2022, the team successfully pitched their idea to the Dashing through the Stars mission, which had issued an open call for project proposals to be carried out on board the ION SCV004 satellite, launched in January 2022. During the autumn of 2022, the team uplinked the code for the program to the satellite already in orbit.
The researchers trained a simple model to detect changes in cloud cover from aerial images directly onboard the satellite, in contrast to training on the ground. The model was based on an approach called few-shot learning, which enables a model to learn the most important features to look for when it has only a few samples to train from. A key advantage is that the data can be compressed into smaller representations, making the model faster and more efficient.
Vít Růžička explained: ‘The model we developed, called RaVAEn, first compresses the large image files into vectors of 128 numbers. During the training phase, the model learns to keep only the informative values in this vector; the ones that relate to the change it is trying to detect (in this case, whether there is a cloud present or not). This results in extremely fast training due to having only a very small classification model to train.’
Whilst the first part of the model, to compress the newly-seen images, was trained on the ground, the second part (which decided whether the image contained clouds or not) was trained directly on the satellite.
Normally, developing a machine learning model would require several rounds of training, using the power of a cluster of linked computers. In contrast, the team’s tiny model completed the training phase (using over 1300 images) in around one and a half seconds.
When the team tested the model’s performance on novel data, it automatically detected whether a cloud was present or not in around a tenth of a second. This involved encoding and analysing a scene equivalent to an area of about 4.8x4.8 km2 area (equivalent to almost 450 football pitches).
According to the researchers, the model could easily be adapted to carry out different tasks, and to use other forms of data. Vít Růžička added: ‘Having achieved this demonstration, we now intend to develop more advanced models that can automatically differentiate between changes of interest (for instance flooding, fires, and deforestation) and natural changes (such as natural changes in leaf colour across the seasons). Another aim is to develop models for more complex data, including images from hyperspectral satellites. This could allow, for instance, the detection of methane leaks, and would have key implications for combatting climate change.’
Performing machine learning in outer space could also help overcome the problem of on-board satellite sensors being affected by the harsh environmental conditions, so that they require regular calibration. Vít Růžička said: ‘Our proposed system could be used in constellations of non-homogeneous satellites, where reliable information from one satellite can be applied to train the rest of the constellation. This could be used, for instance, to recalibrate sensors that have degraded over time or experienced rapid changes in the environment.’
Professor Andrew Markham, who supervised Vít’s DPhil research, said ‘Machine learning has a huge potential for improving remote sensing – the ability to push as much intelligence as possible into satellites will make space-based sensing increasingly autonomous. This would help to overcome the issues with the inherent delays between acquisition and action by allowing the satellite to learn from data on board. Vít’s work serves as an interesting proof-of-principle.’
This project was conducted in collaboration with the European Space Agency (ESA) Φ-lab via the Cognitive Cloud Computing in Space (3CS) campaign and the Trillium Technologies initiative Networked Intelligence in Space (NIO.space) and partners at D-Orbit and Unibap.
Notes for editors:
This work was presented at the International Geoscience and Remote Sensing Symposium (IGARSS) conference on Friday 21 July 2023.
For media enquiries and interview requests, contact Dr Caroline Wood, University of Oxford: caroline.wood@admin.ox.ac.uk 01865 280534 Images are available on request.
About the University of Oxford
Oxford University has been placed number 1 in the Times Higher Education World University Rankings for the seventh year running, and number 2 in the QS World Rankings 2022. At the heart of this success are the twin-pillars of our ground-breaking research and innovation and our distinctive educational offer.
Oxford is world-famous for research and teaching excellence and home to some of the most talented people from across the globe. Our work helps the lives of millions, solving real-world problems through a huge network of partnerships and collaborations. The breadth and interdisciplinary nature of our research alongside our personalised approach to teaching sparks imaginative and inventive insights and solutions.
Through its research commercialisation arm, Oxford University Innovation, Oxford is the highest university patent filer in the UK and is ranked first in the UK for university spinouts, having created more than 200 new companies since 1988. Over a third of these companies have been created in the past three years. The university is a catalyst for prosperity in Oxfordshire and the United Kingdom, contributing £15.7 billion to the UK economy in 2018/19, and supports more than 28,000 full time jobs.
About the European Space Agency
The European Space Agency (ESA) provides Europe’s gateway to space.
ESA is an intergovernmental organisation, created in 1975, with the mission to shape the development of Europe’s space capability and ensure that investment in space delivers benefits to the citizens of Europe and the world
ESA has 22 Member States: Austria, Belgium, the Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Luxembourg, the Netherlands, Norway, Poland, Portugal, Romania, Spain, Sweden, Switzerland and the United Kingdom. Latvia, Lithuania, Slovakia and Slovenia are Associate Members.
ESA has established formal cooperation with four Member States of the EU. Canada takes part in some ESA programmes under a Cooperation Agreement.
By coordinating the financial and intellectual resources of its members, ESA can undertake programmes and activities far beyond the scope of any single European country. It is working in particular with the EU on implementing the Galileo and Copernicus programmes as well as with Eumetsat for the development of meteorological missions.
Learn more about ESA at www.esa.int
Article Title
Fast model inference and training on-board of Satellites