image: (a) MLAO was embedded in the microscope control to improve imaging quality by modulate the illumination light. Right-hand side showed an example of one operation cycle using MLAO. After using MLAO with two sample exposures, the imaging quality was largely improved as the image labelled as corrected was much brighter and sharper than the image labelled as initial. (b) MLAO was used in three-photon intravital deep tissue imaging of a mouse brain. After MLAO, the cells were brighter, sharper and more recognisable. (c) MLAO was used in super-resolution imaging of Drosophila larval neuromuscular junctions. After MLAO, the image was much sharper with a lot more fine features resolved and hexagonal artefacts removed.
Credit: by Qi Hu, Martin Hailstone, Jingyu Wang, Matthew Wincott, Danail Stoychev, Huriye Atilgan, Dalia Gala, Tai Chaiamarit, Richard M. Parton, Jacopo Antonello, Adam M. Packer, Ilan Davis & Martin J. Booth
Microscopes were important tools in a wide range of scientific research fields including biochemistry, physiology and neuroscience. Adaptive optics (AO) has been used in microscopes to correct light wavefront distortions introduced by innate limitations of optical systems and non-uniform structures of imaging samples. Although largely improving the imaging quality of microscopes, AO is hard to control as sample structures and wavefront distortions were unknown prior to imaging. Conventional AO methods often require additional wavefront sensors which can be cumbersome and costly. Proposed sensorless approaches normally require iterative correction processes and thus lead to photo-damage of the samples. Until now, there is no unified method that can be easily implemented and transferred across different microscope modalities.
In the cover article of Light: Science & Application issue 12, a group of researchers from Oxford developed a machine learning-based AO (MLAO) method to incorporate AO in the microscope imaging process and to greatly improve imaging quality. The project was led by Dr Qi Hu, a Schmidt AI in science fellow, and supervised by Prof Martin Booth, both from the department of Engineering Science, University of Oxford. Compared to conventional AO methods, this new method required fewer sample exposures and was robust to a range of challenging imaging tasks including coping with high noise level, random sample motions and ‘blinking’ events in functional imaging. The method was demonstrated in a two-photon, three-photon and widefield three-dimensional structured illumination super-resolution microscope. Different from previous machine learning based approaches, this method encapsulated physical understandings and proposed a bespoke convolutional neural network (CNN) architecture that was orders of magnitude smaller than commonly used other CNN structures. Furthermore, this specially designed CNN embedded physical properties of imaging process. In the paper, the authors stated that “This means that the internal CNN configuration needs no-longer to be considered as a ``black box", but can be used to provide physical insights on internal workings and how information about aberrations is encoded into images.”
1. A universal method that can be implemented in different microscope modalities.
Different from previous methods which normally required specific optical design or were limited to certain microscope modalities, this proposed method was demonstrated to be transferrable to different microscopes. Although different training dataset needs to be generated to incorporate different physical models of imaging systems, the same framework can be followed. CNN architecture was also unchanged when the algorithm was trained for different imaging tasks. In the paper, a two-photon, three-photon and a widefield three-dimensional structured illumination microscope were used for demonstration to show that the method was highly versatile and widely applicable.
2. A more efficient algorithm requires much fewer sample exposures.
A fair comparison was drawn between the MLAO method and its conventionally used counterparts. Statistical results suggested that in most cases, the MLAO method outperformed the conventional methods by correcting better with fewer sample exposures. Only for a small number of cases, the MLAO method performed equivalently to the conventional methods. The MLAO method also performed more consistently and reliably.
3. A more robust algorithm coping with challenging imaging tasks.
The MLAO method was also demonstrated to be effective in challenging imaging tasks such as live imaging with random sample motions and functional dynamic imaging in which conventional methods normally failed. MLAO was also shown to have desirable features of being resilient to variations such as imaging sampling rate, noise level and out-of-focus structures.
4. A non- “black box” algorithm providing physical insights.
The CNN architecture used in the MLAO algorithm was specially designed to encapsulate physical understandings of the imaging process. By drawing links of CNN weights to physical properties, not just MLAO algorithm performances could be examined but its pros and cons to be understood. This unique feature also provides insights for future method design.
With its versatility and effectiveness, the MLAO method has great potential in microscopy imaging applications and can be beneficial for a wide range of science research fields including biochemistry, physiology, ophthalmology and neuroscience.