Revolutionizing crop phenotyping: self-supervised deep learning enhances green fraction estimation in rice and wheat

The accurate measurement of the green fraction (GF), a critical photosynthetic trait in crops, typically relies on RGB image analysis employing segmentation algorithms to identify green pixels within the crop. Traditional methods have limitations in accuracy due to environmental variances, while advanced deep learning techniques, like the SegVeg model, show improvement but don't fully leverage the latest vision transformer models. A significant challenge in applying these state-of-the-art techniques is the lack of comprehensive, annotated datasets for plant phenotyping. Although synthetic image generation offers a partial solution, addressing the realism gap between synthetic and real field images remains a crucial area for future research to enhance the accuracy of GF estimation.

In July 2023, Plant Phenomics published a research article entitled by “Enhancing green fraction estimation in rice and wheat crops: a self-supervised deep learning semantic segmentation approach ”. In this study, the objective was to enhance a self-supervised plant phenotyping pipeline for semantic segmentation of RGB images of rice and wheat, considering their contrasting field backgrounds.

The methodology involved three main steps: (1) Collection of real in situ images and their manual annotations from different sites, and generation of simulated images with labels using the Digital Plant Phenotyping Platform (D3P). (2) Application of the CycleGAN domain adaptation method to minimize the domain gap between the simulated (sim) and real datasets, creating a simulation-to-reality (sim2real) dataset. (3) Evaluation of three deep learning models (U-Net, DeepLabV3+, and SegFormer) trained on real, sim, and sim2real datasets, comparing their performance at pixel and image scales, with a focus on Green Fraction (GF) estimation. The results showed that domain adaptation through CycleGAN effectively bridged the gap between simulated and real images, as evidenced by improved realism in plant textures and soil backgrounds, and a decrease in Euclidean distance between the sim2real and real images. The pixel-scale segmentation demonstrated that U-Net and SegFormer outperformed DeepLabV3+, with SegFormer, especially when trained on the sim2real dataset, exhibiting the highest F1-score and accuracy. This trend was consistent for both rice and wheat crops. The sim2real dataset enabled the best performance in GF estimation, showing close results between simulated and real datasets, especially for wheat. The study also utilized the best-performing model, SegFormer, trained on the sim2real dataset, to explore GF dynamics, effectively capturing the growth stages of rice and wheat, thus indicating accurate GF estimation.  The study also identified critical factors affecting estimation uncertainty, such as nonuniform brightness within images and the presence of senescent leaves. The self-supervised nature of the pipeline, requiring no human labels for training, was emphasized as a significant time-saver in image annotation.

Overall, the research demonstrated that SegFormer trained on the sim2real dataset outperformed other models, highlighting the effectiveness of the self-supervised approach in semantic segmentation for plant phenotyping. The success of this method opens avenues for further research in enhancing the realism of simulated images and applying more sophisticated domain adaptation models for accurate GF estimation throughout the entire crop growth cycle.

###

References

Authors

Yangmingrui Gao^1†, Yinglun Li^1†, Ruibo Jiang¹, Xiaohai Zhan¹, Hao Lu², Wei Guo³, Wanneng Yang⁴, Yanfeng Ding¹, and Shouyang Liu^1*

Affiliations

¹Plant Phenomics Research Centre, Academy for Advanced Interdisciplinary Studies, Jiangsu Collaborative Innovation Center for Modern Crop Production, Nanjing Agricultural University, Nanjing, China.

²Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China.

³Graduate School of Agricultural and Life Sciences, The University of Tokyo, 1-1-1 Midori-cho, Nishitokyo City, Tokyo, Japan.

⁴National Key Laboratory of Crop Genetic Improvement, National Center of Plant Gene Research, and Hubei Key Laboratory of Agricultural Bioinformatics, Huazhong Agricultural University, Wuhan 430070, China.

About Shouyang Liu

He is a professor at the Plant Phenomics Research Centre of Nanjing Agricultural University. His research interests are (a) to develop high throughput phenotyping equipment and algorithms for large fields to tap into high-yielding, high-quality and stress-resistant phenotypic traits of crops; (b) to explore gene-environment interactions mechanisms of key crop traits and to develop three-dimensional mechanistic models; and (c) to integrate multi-source near-earth-remote sensing monitoring and three-dimensional modeling of crops to enhance the prediction capability from genotype to phenotype.