Revolutionizing plant phenotyping: deep learning and 3D point cloud technology in overcoming reconstruction challenges

The 3-dimensional point cloud technology revolutionizes non-invasive measurement of plant phenotypic parameters, offering vital data for agriculture and research. Current research focuses on overcoming the limitations of 2.5D imaging and occlusions. Methods such as structure from motion, multi-view stereo, and advanced active 3D reconstruction techniques are being explored for this purpose. However, issues persist with incomplete data acquisition and the inaccuracy of phenotypic parameter extraction due to plant occlusions and environmental factors. Recently, integrating deep learning with point cloud analysis and innovative techniques like Reinforcement Learning Agent Controlled GAN Network and multi-scale geometry-aware Transformer networks show promise in addressing these challenges by enhancing point cloud completion and maintaining the integrity of the original data.

In November 2023, Plant Phenomics published a research article entitled by “Point Cloud Completion of Plant Leaves under Occlusion Conditions Based on Deep Learning”.

This research utilized neural network-based point cloud completion, specifically the PF-Net algorithm, to reconstruct 3D models of flowering Chinese Cabbage leaves, which are challenging due to their complex structures. A dataset of cabbage leaf point clouds was created and trained using PF-Net, with incomplete point clouds obtained using an Azure Kinect camera. The neural network supplemented these incomplete point clouds, achieving full 3D reconstruction, which was then compared with the MVS-SFM algorithm for accuracy. The results indicated that the PF-Net successfully completed point clouds of various shapes and bending degrees from both MVS-SFM and Azure Kinect datasets, showcasing its strong ability to complete complex structures. However, the completion was less effective in areas with multiple missing sections. As the missing ratio increased, the completion effect diminished, particularly in the holes of the leaf point cloud, suggesting that while the network learned structural relationships effectively, it struggled with larger missing areas. Under natural occlusion conditions, the PF-Net improved the completeness of leaf point clouds compared to the MVS-SFM algorithm, but struggled with multiple occlusions or high missing ratios. The research also examined the reconstruction precision and leaf area extraction results. About half of the point pairs in the reconstructed point clouds had a distance error of less than 2 mm, indicating moderate consistency with the MVS-SFM method. The leaf area extraction analysis revealed that the accuracy of leaf area estimation improved after point cloud completion, with results close to those obtained using the MVS-SFM algorithm.

Despite these promising results, the study acknowledged gaps in completing real-world incomplete point clouds and the challenge of point cloud orientation due to the rotation invariance characteristic of point clouds. Future improvements include developing more realistic training datasets and addressing the orientation estimation challenge, possibly through a multi-stage completion network that can learn and adjust the model's proper pose before completing it. Overall, the research offers a new perspective for plant phenotype studies using active sensors and confirms the potential of deep learning in enhancing phenotypic parameter estimation precision.

###

References

Authors

Haibo  Chen^1,2†, Shengbo  Liu^2,3†, Congyue  Wang^2,3, Chaofeng  Wang^2,3, Kangye  Gong⁴, Yuanhong  Li^2,3*†, and Yubin  Lan^2,3,5*†

†These authors contributed equally to this work.

Affiliations

¹Experimental Basis and Practical Training Center, South China Agricultural University, Guangzhou, China.

²National Center for International Collaboration Research on Precision Agricultural Aviation Pesticides Spraying Technology, Guangzhou, China.

³College of Electronics Engineering (College of Artificial Intelligence), South China Agricultural University, Guangzhou, China.

⁴College of Engineering, South China Agricultural University, Guangzhou, China.

⁵Department of Biological and Agricultural Engineering, Texas A&M University, College Station, TX, United States.

About Yubin  Lan

He is a Professor and doctoral supervisor of South China Agricultural University. He has long been engaged in the development and application research of precision agricultural aviation, aviation pesticide spraying technology and aerial remote sensing technology.