融合多尺度特征的枸杞果实颗粒分割方法

An Instance Segmentation Method for Chinese Wolfberry Fruit Particles Based on Multi-scale Feature Fusion

摘要

精确的果实颗粒形状是评估枸杞生长状态、测量果实尺寸的重要基础。本文提出了一种融合多尺度特征的枸杞果实颗粒分割网络（Chinese Wolfberry Fruitlet Particles Instance Segmentation Network，CWISNet），用于从图像中提取精确的枸杞果实颗粒形状。CWISNet融合Transformer的全局特征提取能力和卷积神经网络的局部细节特征能力提取逐像素语义特征，融合ResNet-34的深层特征提取能力和空洞卷积的大感受野能力生成边缘信息，利用分级融合策略对逐像素语义特征和边缘信息进行融合，生成高质量的逐像素语义特征，最后利用多级候选框提取出精细的枸杞果实颗粒形状。综合考虑形状损失和边界损失构建损失函数，用于对模型进行训练。选择宁夏回族自治区中宁县为实验区，选择Mask R-CNN、YOLACT、HTC、SOLOv2作为对比模型开展实验。实验结果表明，本方法的边界框平均精度均值和掩码平均精度均值在IoU=0.5阈值下分别达到了92.5%和92.3%，明显优于对比模型，在密集遮挡和边界模糊场景下同样表现出了卓越的性能，可为科研和生产实际提供技术和数据支撑。

Abstract

Accurate fruit particle shape is an important basis for evaluating the growth status of Chinese Wolfberry and measuring fruit size. This paper proposes a Chinese Wolfberry Fruitlet Particles Instance Segmentation Network (CWISNet) that fuses multi-scale features to extract accurate shapes of Chinese Wolfberry fruitlet particles from images. CWISNet combines the global feature extraction capability of Transformer with the local detail feature extraction capability of the convolutional neural network to obtain pixel-wise semantic features. It also fuses the deep feature extraction capability of ResNet-34 with the large receptive field capability of dilated convolution to generate edge information. A hierarchical fusion strategy is used to merge the pixel-wise semantic features and edge information, producing high-quality pixel-wise semantic features. Finally, multi-level candidate boxes are used to extract fine Chinese Wolfberry fruitlet particle shapes. Considering both shape loss and boundary loss, CWISNet constructs a loss function to train the model. Zhongning County in Ningxia Hui Autonomous Region is selected as the experimental area, and Mask R-CNN, YOLACT, HTC, and SOLOv2 are used as comparative models for the experiments. The results show that the proposed method achieves a box mAP of 92.5% and a mask mAP of 92.3% at an IoU threshold of 0.5, outperforming the comparative models by a clear margin. It also demonstrates excellent performance in scenes with dense occlusions and blurred boundaries, providing technical and data support for both research and field applications.

关键词

枸杞 / 实例分割 / 边缘检测 / 混合架构 / 深度学习

Key words

Chinese Wolfberry / instance segmentation / edge detection / hybrid architecture / deep learning

参考文献

[1] Dai G, Fan J, Dewi C. ITF-WPI: Image and text based cross-modal feature fusion model for wolfberry pest recognition[J]. Computers and Electronics in Agriculture, 2023, 212: 108129.

[2] Xing Z, Wang Y, Qu A, et al. MFENet: Multi-scale feature extraction network for images deblurring and segmentation of swinging wolfberry branch[J]. Computers and Electronics in Agriculture, 2023, 215: 108413.

[3] Wang J, Tan D, Sui L, et al. Wolfberry recognition and picking-point localization technology in natural environments based on improved Yolov8n-Pose-LBD[J]. Computers and Electronics in Agriculture, 2024, 227: 109551.

[4] Gongal A, Amatya S, Karkee M, et al. Sensors and systems for fruit detection and localization: A review[J]. Computers and Electronics in Agriculture, 2015, 116: 8-19.

[5] 苗中华, 沈一筹, 王小华, 等. 自然环境下重叠果实图像识别算法与试验[J]. 农业机械学报, 2016, 47(06): 21-26.

[6] 陈浩, 任奕林. 大田经济作物性状研究中的图像分割技术应用[J]. 广东农业科学, 2021, 48(11): 123-135.

[7] Tian Y, Yang G, Wang Z, et al. Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J]. Computers and Electronics in Agriculture, 2019, 157: 417-426.

[8] Lv J, Xu H, Han Y, et al. A visual identification method for the apple growth forms in the orchard[J]. Computers and Electronics in Agriculture, 2022, 197: 106954.

[9] Wang C, Wang H, Han Q, et al. Strawberry detection and ripeness classification using yolov8+ model and image processing method[J]. Agriculture, 2024, 14(5): 751.

[10] Wan S, Goudos S. Faster R-CNN for multi-class fruit detection using a robotic vision system[J]. Computer Networks, 2020, 168: 107036.

[11] Shen L, Su J, Huang R, et al. Fusing attention mechanism with Mask R-CNN for instance segmentation of grape cluster in the field[J]. Frontiers in Plant Science, 2022, 13: 934450.

[12] 姚青, 谷嘉乐, 吕军, 等. 改进RetinaNet的水稻冠层害虫为害状自动检测模型[J]. 农业工程学报, 2020, 36(15): 182-188.

[13] 王梁, 侯义锋, 贺杰. 基于Mask-RCNN的自然场景下油茶果目标识别与检测[J]. 中国农机化学报, 2022, 43(12): 148-154.

[14] Huang X, Peng D, Qi H, et al. Detection and instance segmentation of grape clusters in orchard environments using an improved mask R-CNN model[J]. Agriculture, 2024, 14(6): 918.

[15] Yu Y, Zhang K, Yang L, et al. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J]. Computers and Electronics in Agriculture, 2019, 163: 104846.

[16] Jia W, Tian Y, Luo R, et al. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot[J]. Computers and Electronics in Agriculture, 2020, 172: 105380.

[17] Huang Y, Wang T, Basanta H. Using fuzzy mask R-CNN model to automatically identify tomato ripeness[J]. IEEE Access, 2020, 8: 207672-207682.

[18] Liu Y, Sun P, Wergeles N, et al. A Survey and Performance Evaluation of Deep Learning Methods for Small Object Detection[J]. Expert Systems with Applications, 2021, 172: 114602.

[19] Cong P, Li S, Zhou J, et al. Research on instance segmentation algorithm of greenhouse sweet pepper detection based on improved mask RCNN[J]. Agronomy, 2023, 13(1): 196.

[20] Liu H, Wang X, Zhao F, et al. Upgrading swin-B transformer-based model for accurately identifying ripe strawberries by coupling task-aligned one-stage object detection mechanism[J]. Computers and Electronics in Agriculture, 2024, 218: 108674.

[21] Xiao B, Nguyen M, Yan W. Fruit ripeness identification using transformers[J]. Applied Intelligence, 2023, 53(19): 22488-22499.

[22] 陈鸿坤, 罗会兰. 多尺度语义信息融合的目标检测[J]. 电子与信息学报, 2021, 43(07): 2087-2095.

[23] 郑剑, 贺朝辉, 于祥春. 基于相邻特征融合与特征解耦的一阶段目标检测[J]. 北京航空航天大学学报, 2025, 51(04): 1205-1214.

[24] Liu G, Zhang Y, Liu J, et al. An improved YOLOv7 model based on Swin Transformer and Trident Pyramid Networks for accurate tomato detection[J]. Frontiers in Plant Science, 2024, 15: 1452821.

[25] Charisis C, Argyropoulos D. Deep learning-based instance segmentation architectures in agriculture: A review of the scopes and challenges[J]. Smart Agricultural Technology, 2024, 8: 100448.

[26] Yang J, Jiao L, Shang R, et al. EPT-Net: Edge perception transformer for 3d medical image segmentation[J]. IEEE Transactions on Medical Imaging, 2023, 42(11): 3229-3243.

[27] He C, Li S, Xiong D, et al. Remote sensing image semantic segmentation based on edge information guidance[J]. Remote Sensing, 2020, 12(9): 1501.

[28] Sheng X, Kang C, Zheng J, et al. An edge-guided method to fruit segmentation in complex environments[J]. Computers and Electronics in Agriculture, 2023, 208: 107788.

[29] Jia W, Cao K, Liu M, et al. FCOS-EAM: An accurate segmentation method for overlapping green fruits[J]. Computers and Electronics in Agriculture, 2024, 226: 109392.

[30] Zimmermann R S, Siems J N. Faster training of Mask R-CNN by focusing on instance boundaries[J]. Computer Vision and Image Understanding, 2019, 188: 102795.

[31] Du W, Liu P. Instance segmentation and berry counting of table grape before thinning based on AS-SwinT[J]. Plant Phenomics, 2023, 5: 0085.

[32] Li H, Xiong X, Liu C, et al. SFFNet: Staged feature fusion network of connecting convolutional neural networks and graph convolutional neural networks for hyperspectral image classification[J]. Applied Sciences, 2024, 14(6): 2327.

[33] Pei F, Li Z, Mei S, et al. Recognition of Chinese wolfberry images with windy and sandy noises using improved YOLOv8[J]. International Journal of Agriculture and Biological Engineering, 2025, 18(2): 239-247.

[34] Wang R, Tan D, Ju X, et al. An object feature-based recognition and localization method for wolfberry[J]. Sensors, 2025, 25(11): 3365.

[35] Micikevicius P, Narang S, Alben J, et al. Mixed precision training[J]. arXiv preprint arXiv: 1710.03740, 2017.

基金资助

宁夏回族自治区自然科学基金(2024AAC03419)

山东省气象局创新团队专项(2024sdcxtd04)

风云卫星应用先行计划（三期）项目(FY-APP-2024.0301)

中央引导地方科技发展专项项目(2024FRD05056)

山东省自然科学基金(ZR2021MD097)