基于对比学习的类别不均衡番茄叶片病虫害图像-文本检索方法研究

Research on Image-Text Retrieval Method for Tomato Leaf Diseases and Pests with Class Imbalance Based on Contrastive Learning

摘要

番茄作为重要的经济作物，其病虫害防治对保障农业生产效益至关重要。番茄叶片病虫害图像能够直观展示病害形态特征和分布情况，但存在因分辨率有限、遮挡等情况导致细节丢失的缺点，而文本能够提供详细的发病症状描述、病因分析、防治策略，进而可以弥补图像的不足。然而，图像和文本两种模态存在语义鸿沟，使得利用图像精确检索相应文本描述困难，利用文本描述精确检索相应图像同样极具挑战性。此外，实际番茄叶片病虫害数据常存在类别不均衡问题，导致模型对主流类别过拟合、对稀有类别欠拟合。为解决上述问题，本文提出了一种基于对比学习的番茄叶片病虫害图像-文本跨模态检索方法并构建了首个中文番茄病虫害图像-文本数据集，其中模拟了实际生产中类别不均衡现象（如最大类别样本数为最小类别的近9倍），帮助研究更贴近实际生产中的挑战。为实现图像与文本模态的精确对齐，设计了一种基于三元组的对比学习方法，引入双曲空间建模层次语义关系，以拉近同类特征距离并拉远异类特征距离。提出自适应分类损失函数，动态调节模型对不同类别的关注度，有效缓解类别不均衡对检索性能的影响。此外，为提取局部病斑特征，设计了一种基于预训练编码器的参数冻结迁移学习模块，通过冻结编码器的参数来提取细粒度语义特征，以避免从头训练导致的额外训练成本。在实验部分，我们将所提出方法与先进的检索方法CCA、DSCMR、SCH、DDBH 和DScPH 进行对比，在图像检索文本任务上分别提升了28.68%、9.58%、3.38%、1.76%、1.03%，在文本检索图像任务上分别提升了35.71%、6.19%、0.94%、1.05%、0.54%。此外，为验证所提方法在图像编码器架构选择上的有效性，使用不同的图像特征编码器VGG16、MobileNet V2、CLIP-ViT-B/32 进行对比，平均性能分别提升了12.965%、1.45%、1.005%。

Abstract

As an important cash crop, effective pest and disease control in tomatoes plays a crucial role in ensuring agricultural production efficiency. Images of tomato leaf pests and diseases can visually display the morphological characteristics and distribution patterns of the conditions, but they have drawbacks such as the loss of detail due to limited resolution, occlusion, and other factors. In contrast, text can provide detailed descriptions of disease symptoms, etiological analysis, and prevention strategies, thereby compensating for the deficiencies of images. However, a semantic gap exists between images and text, which makes it difficult to accurately retrieve corresponding text descriptions from images and vice versa. In addition, real-world tomato leaf disease and pest data often exhibit class imbalance, which causes models to overfit to dominant classes and underfit to rare classes. To address the above issues, this study proposes a contrastive learning-based image-text cross-modal retrieval method for tomato leaf disease and pest, and constructs the first Chinese tomato disease and pest image-text dataset. The dataset simulates the class imbalance observed in practical production (e.g., the number of samples in the largest class is nearly 9 times that in the smallest class), ensuring the research better reflects real-world challenges. To achieve accurate alignment between image and text modalities, this study designs a triplet-based contrastive learning method, which introduces hyperbolic space to model hierarchical semantic relationships, thereby pulling features of the same class closer while pushing features of different classes apart. Additionally, this study proposes an adaptive classification loss function to dynamically adjust the model's attention to different classes, effectively mitigating the impact of class imbalance on retrieval performance. Furthermore, to extract local lesion features, this study designs a parameter-freezing transfer learning module based on pre-trained encoders. The module extracts fine-grained semantic features by freezing the encoder parameters, thereby avoiding additional training costs associated with training from scratch. In the experiment, this study compares the proposed method with advanced retrieval methods ( CCA, DSCMR, SCH, DDBH, and DScPH). The proposed method achieves improvements of 28.68%, 9.58%, 3.38%, 1.76%, and 1.03% in the image-to-text retrieval tasks, and 35.71%, 6.19%, 0.94%, 1.05%, and 0.54% in the text-to-image retrieval tasks, respectively. In addition, to verify the effectiveness of the proposed method in selecting image encoder architectures, this study conducts comparisons using different image feature encoders (VGG16, MobileNet V2, and CLIP-ViT-B/32), with the average performance improvements of 12.965%, 1.45%, and 1.005%, respectively.

关键词

番茄病虫害识别 / 图像-文本检索 / 深度学习 / 对比学习

Key words

Tomato disease and pest identification / image-text retrieval / deep learning / contrastive learning

参考文献

[1] 郭小清, 范涛杰, 舒欣. 基于改进 Multi-Scale AlexNet 的番茄叶部病害图像识别[J]. 农业工程学报, 2019, 35(13): 162-169.

[2] 柴帅, 李壮举. 基于迁移学习的番茄病虫害检测[J]. 计算机工程与设计, 2019, 40(06): 1701-1705.

[3] 陶兆胜, 石鑫宇, 王勇, 等. 基于改进YOLOv5s的番茄叶片病害检测方法[J]. 沈阳农业大学学报, 2023, 54(06): 712-721.

[4] 刘立波, 赵斐斐. 融合注意力机制的枸杞虫害图文跨模态检索方法[J]. 农业机械学报, 2022, 53(02): 299-308.

[5] 宋文韬, 姜茹月, 舒欣. 基于零样本学习的枸杞虫害识别[J]. 江苏农业学报, 2024, 40(02): 320-330.

[6] Dai G, Fan J, Dewi C . ITF-WPI: Image and text based cross-modal feature fusion model for wolfberry pest recognition[J]. Computers and Electronics in Agriculture, 2023, 212: 108129.

[7] Xu J, Zhou H, Hu Y, et al. High-accuracy tomato leaf disease image-text retrieval method utilizing LAFANet[J]. Plants, 2024, 13(9): 1176.

[8] Feng X, Zhao C, Wang C, et al. A vegetable leaf disease identification model based on image-text cross-modal feature fusion[J]. Frontiers in Plant Science, 2022, 13: 918940.

[9] 杨筝, 程云志, 常开心, 等. 基于细粒度特征增强交互网络的植物病虫害识别[J]. 河南大学学报(自然科学版), 2024, 54(06): 722-729.

[10] 陈磊, 刘立波, 王晓丽. 2020年宁夏枸杞虫害图文跨模态检索数据集[J]. 中国科学数据(中英文网络版), 2022, 7(03): 149-156.

[11] 聂葳, 叶成炜, 杨家慧, 等. 基于CHINESE-CLIP跨模态图像文本检索研究[J]. 电子制作, 2024, 32(22): 61-66.

[12] 张北辰, 李亮, 查正军, 等. 基于跨模态对比学习的视觉问答主动学习方法[J]. 计算机学报, 2022, 45(08): 1730-1745.

[13] 周坤, 徐黎明, 郑伯川, 等. 自适应高效深度跨模态增量哈希检索算法[J]. 计算机工程与应用, 2023, 59(02): 85-93.

[14] 邢嘉璐, 刘建平, 周国民, 等. 基于参数高效微调的跨模态枸杞虫害识别模型D-PAG[J]. 农业大数据学报, 2024, 6(04): 509-521.

[15] Wei Y, Liu X, An D, et al. Citrus diseases and pests image-text retrieval based on multi-modal transformer [C]. 2023 International Conference on High Performance Big Data and Intelligent Systems (HDIS). IEEE, 2023: 66-70.

[16] 黄溪, 王先兵, 林海, 等 . 基于自集成视觉Transformer的图像检索[J]. 武汉大学学报(工学版), 2024, 57(12): 1795-1802.

[17] 姜飞, 叶炜, 李兆星, 等. 基于特征交互的样本不均衡的玉米病害检测方法[J]. 华南农业大学学报, 2025, 46(03): 399-406.

[18] 孙宝刚, 何国斌. 不均衡少标签样本下基于语义自动编码网络的高光谱图像分类[J]. 红外技术, 2025, 47(04): 429-436.

[19] 郑宗生, 霍志俊, 高萌, 等 . 基于类中心优化辅助三元组损失的遥感图像检索[J]. 计算机工程, 2025, 51(05): 305-313.

[20] 代佳洋, 周栋. 基于多任务学习的跨语言信息检索方法研究[J]. 广西师范大学学报(自然科学版), 2022, 40(06): 69-81.

[21] 尚皓玺, 郭小燕, 朱恒宇. 基于迁移学习与GhostNet模型的农业害虫图像识别研究[J]. 软件导刊, 2022, 21(11): 137-143.

[22] 王丽妍. 基于迁移学习的智慧农业病虫害图像识别方法[J]. 农业工程技术, 2024, 44(02): 120-121.

[23] 万鹏, 赵竣威, 朱明, 等. 基于改进Res Net50 模型的大宗淡水鱼种类识别方法[J]. 农业工程学报, 2021, 37(12): 159-168.

[24] Sun C, Latapie H, Liu G, et al. Deep normalized cross-modal hashing with bi-direction relation reasoning[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 4941-4949.

基金资助

国家自然科学基金青年基金项目(62202281)