基于Transformer的半监督偏多标签图像分类方法

Transformer-Based Image Classification Method of Semi-Supervised Partial Multi-Label

摘要

针对在半监督偏多标签学习(SSPML: Semi-Supervised Partial Multi-Label Learning)方法中, 特征和标签空间之间以及各自内部存在复杂的相关性, 且利用标签相关性构建模型时, 容易受到候选标签集中噪声标签干扰的问题, 基于Transformer提出一种新的半监督偏多标签图像分类算法T-SSPML(Transformer Encoder-Based Semi-Supervised Partial Multi-Label Learning)。利用Transformer编码器构建主干网络, 将图像和标签特征共同作为输入, 并定义新的损失函数, 利用多头自注意力机制学习整体性结构信息以及标签间的关联信息。同时采用标签掩码机制, 提升模型学习标签关联信息的能力。通过在多个数据集上实验结果表明, T-SSPML算法相比其他对比算法在绝大多数情况下都取得了更好的分类性能。

Abstract

In SSPML(Semi-Supervised Partial Multi-Label Learning), complex correlations exist between the feature and label, and within them. Models leveraging label correlations are susceptible to interference from noisy labels in candidate label sets. T-SSPML(Transformer Encoder-Based Semi-Supervised Partial Multi-Label Learning), a novel semi-supervised partial multi-label image classification algorithm is proposed based on Transformer. A backbone network is constructed using Transformer encoders, taking both image features and label features as joint inputs, and a new loss function is defined. The multi-head self-attention mechanism is employed to capture holistic structural information and label interdependencies. A label masking mechanism is introduced to enhance the model's capability in learning label correlation information. Extensive experiments on multiple datasets demonstrate that the T-SSPML algorithm achieves superior classification performance compared to other representative methods in most cases.

关键词

偏多标签学习 / 半监督学习 / 图像分类 / 一致性正则化 / 伪标签方法 / Transformer编码器

Key words

partial multi-label learning / semi-supervised learning / image classification / consistency regularization / pseudo-labeling methods / Transformer encoder

参考文献

[1] ZHANG M L, ZHOU Z H. A Review on Multi-Label Learning Algorithms[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8): 1819-1837.

[2] LI Q, PENG X J, QIAO Y, et al. Learning Label Correlations for Multi-Label Image Recognition with Graph Networks[J]. Pattern Recognition Letters, 2020, 138: 378-384.

[3] LI X M, OUYANG J H, ZHOU X T. Centroid Prior Topic Model for Multi-Label Classification[J]. Pattern Recognition Letters, 2015, 62: 8-13.

[4] HE Z Y, LYU P, WU J. Minimum Classification Error Rate Training of Supervised Topic Mixture Model for Multi-Label Text Categorization[C]// 2014 9th International Symposium on Chinese Spoken Language Processing. Piscataway, NJ, USA: IEEE, 2014: 39-43.

[5] ZHANG M L, ZHOU Z H. Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization[J]. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(10): 1338-1351.

[6] XIE M K, HUANG S J. Partial Multi-Label Learning with Noisy Label Identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3676-3687.

[7] XIE M K, HUANG S J. Semi-Supervised Partial Multi-Label Learning[C]// Industrial Conference on Data Mining. Piscataway, NJ, USA: IEEE Publisher, 2020: 691-700.

[8] GUAN Y Y, ZHANG B X, LI W H, et al. Semi-Supervised Partial Multi-Label Classification with Low-Rank and Manifold Constraints[J]. Pattern Recognition Letters, 2021, 151: 112-119.

[9] TAN A H, LIANG J Y, WU W Z, et al. Semi-Supervised Partial Multi-Label Classification via Consistency Learning[J/OL]. Pattern Recognition, 2022, 131(1): 108839[2025-07-25]. https://doi.org/10.1016/j.patcog.2022.108839.

[10] FANG J P, ZHANG M L. Partial Multi-Label Learning via Credible Label Elicitation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(10): 3518-3525.

[11] YUAN J, CHEN S K, ZHANG Y, et al. Graph Attention Transformer Network for Multi-Label Image Classification[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2023, 19(4): 1-16.

[12] WANG Z W, LIU F H, HAN M J, et al. PML-ED: A Method of Partial Multi-Label Learning by Using Encoder-Decoder Framework and Exploring Label Correlation[J/OL]. Information Sciences, 2024, 661: 120165[2025-07-25]. https://doi.org/10.1016/j.ins.2024.120165.

[13] ZHAN W, ZHANG M L. Inductive Semi-Supervised Multi-Label Learning with Co-Training[C]// Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: ACM, 2017: 1305-1314.

[14] LI Q M, WU X M, LIU H, et al. Label Efficient Semi-Supervised Learning via Graph Filtering[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2019: 9582-9591.

[15] LANCHANTIN J, WANG T L, ORDONEZ V, et al. General Multi-Label Image Classification with Transformers[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2021: 16478-16488.

[16] TAN A H, WU W Z. Partial Multi-Label Learning via Semi-Supervised Subspace Collaboration[J/OL]. Knowledge-Based Systems, 2024, 287: 111444[2025-07-25]. https://doi.org/10.1016/j.knosys.2024.111444.

[17] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C]// Advances in Neural Information Processing Systems. New York, NY, USA: Curran Associates, 2017: 5998-6008.

[18] MIYATO T, MAEDA S I, KOYAMA M, et al. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(8): 1979-1993.

[19] CUBUK E D, ZOPH B, SHLENS J, et al. RandAugment: Practical Automated Data Augmentation with a Reduced Search Space[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2020: 3008-3017.

[20] CUBUK E D, ZOPH B, MANE D, et al. AutoAugment: Learning Augmentation Strategies from Data[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2019: 113-123.

[21] LEE D H. Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks[C]// Workshop on Challenges in Representation Learning, ICML. New York, NY, USA: PMLR Publisher, 2013: 1-6.

[22] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: Common Objects in Context[C]// European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.

[23] EVERINGHAM M, VAN GOOL L, WILLIAMS C K I, et al. The Pascal Visual Object Classes Challenge 2007[J]. International Journal of Computer Vision, 2010, 88(2): 303-338.

[24] HUISKES M J, LEW M S. The MIR Flickr Retrieval Evaluation[C]// The 1st ACM International Conference on Multimedia Information Retrieval. New York, NY, USA: ACM Press, 2008: 39-43.

[25] LIU Z W, LUO P, WANG X G, et al. Deep Learning Face Attributes in the Wild[C]// 2015 IEEE International Conference on Computer Vision. Piscataway, NJ, USA: IEEE, 2015: 3730-3738.

[26] XU N, LIU Y P, GENG X. Partial Multi-Label Learning with Label Distribution[C]// AAAI Conference on Artificial Intelligence. Menlo Park, CA, USA: AAAI, 2020: 6510-6517.

[27] KONG X N, NG M K, ZHOU Z H. Transductive Multilabel Learning via Label Set Propagation[J]. IEEE Transactions on Knowledge and Data Engineering, 2013, 25(3): 704-719.

[28] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ, USA: IEEE, 2016: 770-778.

基金资助

国家自然科学基金资助项目(61976141)