基于分层动态掩码视觉 Transformer 的长尾图像识别研究

汪世豪 ,  毛煜 ,  林耀进

南京师范大学学报(工程技术版) ›› 2026, Vol. 26 ›› Issue (3) : 22 -30.

PDF (2207KB)
南京师范大学学报(工程技术版) ›› 2026, Vol. 26 ›› Issue (3) : 22 -30. DOI: 10.3969/j.issn.1672-1292.2026.03.003
计算机科学与技术

基于分层动态掩码视觉 Transformer 的长尾图像识别研究

作者信息 +

Hierarchical Dynamic Masked Vision Transformer for Long-Tail Image Recognition

Author information +
文章历史 +
PDF (2259K)

摘要

现实世界中的许多分类任务天然具有层次化语义结构,当这些层次化分类任务同时面临长尾分布时,识别难度进一步加剧. 现有的长尾学习方法大多将类别视为扁平结构,未能充分利用层次标签中固有的语义关联. 传统的层次分类方法通常假设类别分布均衡,对长尾分布缺乏鲁棒性. 为解决这一问题,本文提出了一种基于分层动态掩码视觉 Transformer 的长尾图像识别算法(hierarchical dynamic masked vision transformer,HDMViT),针对标签分布极不均衡问题,对数据集的父类层标签进行了重新构建以优化层次结构的表达能力. 同时,在视觉 Transformer 模型中引入层次化双重分类头结构,两个分类头均基于 Transformer 编码器输出的同一特征向量进行决策:粗粒度分类头(父类)通过全连接层直接建模父类分布,细粒度分类头(子类)则进一步设计动态 Logit 掩码机制来过滤无关特征干扰,确保模型能够更专注地学习目标类别的关键特征,从而更有效地利用细粒度和粗粒度的语义信息. 实验结果表明,该方法在多个长尾数据集上均表现优异.

Abstract

Many classification tasks in the real world inherently possess hierarchical semantic structures. When these hierarchical classification tasks simultaneously face long-tailed distributions, the difficulty of recognition is further exacerbated. Existing long-tailed learning methods mostly treat categories as flat structures and fail to fully utilize the inherent semantic correlations in hierarchical labels. Traditional hierarchical classification methods, on the other hand, usually assume balanced category distributions and lack robustness to long-tailed distributions. To address this problem, this paper proposes a long-tailed image recognition algorithm based on a Hierarchical Dynamic Masked Vision Transformer (HDMViT). Aiming at the extremely imbalanced label distribution, the parent-level labels of the dataset are reconstructed to enhance the expressive capability of the hierarchical structure. Proposed methodology reconstructs parent-level labels to strengthen hierarchical representation, specifically targeting extreme label imbalance. Meanwhile, a hierarchical dual-classification-head structure is introduced into the Vision Transformer model, in which both classification heads make decisions based on the same feature vector output by the Transformer encoder: the coarse-grained (parent-level) head directly models the parent-class distribution through fully connected layers, while the fine-grained (child-level) head further incorporates a dynamic logit masking mechanism to filter out irrelevant feature interference, ensuring that the model learns the key features of the target classes in a more focused manner, thereby making more effective use of both fine-grained and coarse-grained semantic information. Experimental results show that the proposed method achieves excellent performance on multiple long-tailed datasets.

关键词

深度学习 / 长尾学习 / 不平衡数据 / 视觉 Transformer / 层次分类

Key words

deep learning / long-tail learning / imbalanced data / vision transformer / hierarchical classification

引用本文

引用格式 ▾
汪世豪,毛煜,林耀进. 基于分层动态掩码视觉 Transformer 的长尾图像识别研究[J]. 南京师范大学学报(工程技术版), 2026, 26(3): 22-30 DOI:10.3969/j.issn.1672-1292.2026.03.003

登录浏览全文

4963

注册一个新账户 忘记密码

参考文献

[1]

石泽男, 陈海鹏, 张冬, 等. 预训练驱动的多模态边界感知视觉 Transformer[J]. 软件学报, 2022, 34(5): 2051-2067.

[2]

Li MK, Cheung YM, Lu Y. Long-tailed visual recognition via gaussian clouded logit adjustment[C] // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022.

[3]

Li J, Tan ZC, Wan J, et al.. Nested collaborative learning for long-tailed visual recognition[C] // Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022.

[4]

韩佳艺, 刘建伟, 陈德华, 等. 深度长尾学习研究综述[J]. 自动化学报, 2024, 51(5): 985-1020.

[5]

Dablain D, Krawczyk B, Chawla NV. DeepSMOTE: Fusing deep learning and SMOTE for imbalanced data[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(9): 6390-6404.

[6]

Goodfellow I, Pouget-Abadie J, Mirza M, et al.. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.

[7]

Park S, Lim J, Jeon Y, et al.. Influence-balanced loss for imbalanced visual classification[C] // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Montreal, QC, Canada: IEEE, 2021.

[8]

Zhou YX, Qu Y, Xu X, et al.. ImbSAM: A closer look at sharpness-aware minimization in class-imbalanced recognition[C] // Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Paris, France: IEEE, 2023.

[9]

Ren JW, Yu CJ, Sheng SN, et al.. Balanced meta-Softmax for long-tailed visual recognition[J]. Advances in Neural Information Processing Systems, 2020, 33: 4175-4186.

[10]

Rangwani H, Mondal P, Mishra M, et al.. DeiT-LT: Distillation strikes back for vision transformer training on long-tailed dataset[C] // Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2024.

[11]

Cao KD, Wei C, Gaidon A, et al.. Learning imbalanced datasets with label-distribution-aware margin loss[C] // Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver BC, Canada: NIPS, 2019.

[12]

Kang BY, Xie SN, Rohrbach M, et al.. Decoupling representation and classifier for long-tailed recognition[PP/OL]. V2. arXiv (2020-02-19)[2025-08-02]. https://doi.org/10.48550/arXiv.1910.09217.

[13]

Chen T, Kornblith S, Norouzi M, et al.. A simple framework for contrastive learning of visual representations[C] // Proceedings of the 2020 International Conference on Machine Learning. PmLR, 2020.

[14]

He KM, Fan HQ, Wu YX, et al.. Momentum contrast for unsupervised visual representation learning[C] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA: IEEE, 2020.

[15]

Lin TY, Goyal P, Girshick R, et al.. Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 42(2): 318-327.

[16]

Xu ZZ, Liu RK, Yang S, et al.. Learning imbalanced data with ViTs[C] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, BC, Canada: IEEE, 2023.

[17]

Liu ZW, Miao ZQ, Zhan XH, et al.. Large-scale long-tailed recognition in an open world[C] // Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019.

[18]

Van Horn G, MacAodha O, Song Y, et al. The inaturalist species classification and detection dataset[C] // Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018.

[19]

Zhou BL, Lapedriza A, Khosla A, et al.. Places: A 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(6): 1452-1464.

[20]

Liu Z, Lin YT, Cao Y, et al.. Swin transformer: Hierarchical vision transformer using shifted windows[C] // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Montreal, QC, Canada: IEEE, 2021.

基金资助

国家自然科学基金资助项目(62576158)

福建省自然科学基金重点项目(2026J002035)

福建省自然科学基金面上项目(2026J001987)

AI Summary AI Mindmap
PDF (2207KB)

0

访问

0

被引

详细

导航
相关文章

AI思维导图

/

〈 〉