多频思维链推理的情景式小样本视觉问答方法

Episodic Few-Shot Visual Question Answering Method with Multi-Frequency Chain-of-Thought Reasoning

摘要

现有情景式小样本视觉问答方法多依赖于空间域视觉特征，忽略了频域信息在解耦全局与局部语义中的潜力，导致模型对图像内容的理解层次不足。此外，传统推理机制难以实现跨频率的渐进式交互，限制了跨模态推理能力的进一步提升。针对情景式小样本视觉问答任务，提出了一种基于多频思维链的跨模态推理网络。该方法通过频率域信息对图像内容进行深度过滤，设计自适应高低频分离模块解耦视觉模态中不同层次的语义表达，并构建多模态高低频思维链推理机制，利用认知银行存储跨模态推理中间结果，结合双路径思维链充分挖掘过滤后的视觉信息，以实现更准确的情景式小样本视觉问答。实验表明，该方法显著增强了模型对小样本视觉问答任务的处理能力，提升了跨模态推理的准确性和泛化性能，在公开数据集上的实验结果优于同类算法。

Abstract

To address the challenges of episodic few-shot visual question answering, we propose a Frequency-based Cross-modal Chain-of-Thought Inference Network (FC-CoT-In). The method adaptively filters image content through frequency-domain information, designs an Adaptive High-Low Frequency Separation Module (AHLS) to decouple semantic representations at different levels in the visual modality, and constructs a multimodal high-low frequency chain-of-thought reasoning mechanism. By leveraging a cognitive bank to store intermediate results of cross-modal reasoning and combining a dual-path chain of thought, the approach fully exploits filtered visual information to achieve more accurate episodic few-shot visual question answering. Experiments demonstrate that the proposed method significantly enhances the model's capability in handling Episodic Few-shot Visual Question Answering (EFSVQA) tasks, improves the accuracy and generalization performance of cross-modal reasoning, and outperforms comparable algorithms on public benchmarks.

关键词

情景式小样本视觉问答 / 跨模态推理网络 / 自适应高低频分离模块 / 思维链 / 双路径

Key words

EFSVQA / cross-modal reason network / AHLS / CoT / dual path

参考文献

[1] ANTOL S, AGRAWAL A, LU J, et al. Vqa: Visual question answering[C]//Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015:2425-2433.

[2] SHIH K J, SINGH S, HOIEM D. Where to look: Focus regions for visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Cancun, Mexico: IEEE, 2016:4613-4621.

[3] QIAO T, DONG J, XU D. Exploring human—like attention supervision in visual question answering[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2018:2794-2800.

[4] ZHOU Y, REN T, ZHU C, et al. Trar: Routing the attention spans in transformer for visual question answering[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021:2074-2084.

[5] ANDERSON P, HE X, BUEHLER C, et al. Bottom—up and top—down attention for image captioning and visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018:6077-6086.

[6] CHOWDHURY S, SONI B. Envqa: Improving visual question answering model by enriching the visual feature[J].Engineering Applications of Artificial Intelligence, 2025, 142:109948.

[7] HE K, CHEN X, XIE S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA: IEEE, 2022:16000-16009.

[8] YANG Z, HE X, GAO J, et al. Stacked attention networks for image question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Cancun, Mexico: IEEE, 2016:21-29.

[9] XU H, SAENKO K. Ask, attend and answer: Exploring question—guided spatial attention for visual question answering[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2016:451-466.

[10] KIM J H, JUN J, ZHANG B T. Bilinear attention networks[C]//Neural Information Processing Systems. Montreal, Canada: Curran Associates Inc,2018.

[11] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in neural informa—tion processing systems. Long Beach, CA, USA: Curran Associates Inc, 2017.

[12] YU Z, YU J, CUI Y, et al. Deep modular co—attention networks for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019:6281-6290.

[13] GU M, ZHAO Z, JIN W, et al. Video dialog via multi—grained convolutional self—attention context multi—modal networks[J].IEEE Transactions on Circuits and Sys—tems for Video Technology, 2019, 30(12):4453-4466.

[14] NGUYEN B X, DO T, TRAN H, et al. Coarse—to—fine reas—oning for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022:4558-4566.

[15] 李耀, 李梅. 基于多尺度 WideResNet 的铁轨缺陷小样本检测算法 [J].华东理工大学学报 (自然科学版), 2025, 51(6):843-849.

[16] 全志楠, 林家骏. 文本无关的小样本手写汉字笔迹鉴别方法 [J].华东理工大学学报 (自然科学版), 2018, 44(6):882-886.

[17] 沈震宇, 朱昌明, 王喆. 基于 MAML 算法的 YOLOv3 目标检测模型 [J].华东理工大学学报 (自然科学版), 2022, 48(1):112-119.

[18] WANG W, BAO H, DONG L, et al. Image as a foreign language: Beit pretraining for vision and vision—language tasks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023:19175-19186.

[19] SNELL J, SWERSKY K, ZEMEL R. Prototypical net—works for few—shot learning[C]//31st International Confer—ence on Neural Information Processing Systems. Long Beach, California, USA: Curran Associates Inc, 2017.

[20] ZHANG C, CAI Y, LIN G, et al. DeepEMD: Few—shot image classification with differentiable earth mover's distance and structured classifiers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020:12203-12213.

[21] XIE J, LONG F, LYU J, et al. Joint distribution matters: Deep brownian distance covariance for few—shot classifica—tion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022:7972-7981.

[22] WU J, WANG S, SUN J. AMMD: Attentive maximum mean discrepancy for few—shot image classification[J].Pat—tern Recognition, 2024, 155:110680.

[23] WU H, YE G, ZHOU Z, et al. Dual—view data hallucina—tion with semantic relation guidance for few—shot image recognition[J].IEEE Transactions on Multimedia, 2024, 26:9332-9344.

[24] ZHANG M, ZHANG J, LU Z, et al. IEPT: Instance—level and episode—level pretext tasks for few—shot learning[C]//International Conference on Learning Representations. Vir—tual: ICLR, 2021:16482-16497.

[25] YE H J, HU H, ZHAN D C, et al. Few—shot learning via embedding adaptation with set—to—set functions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020:8808-8817.

[26] LI W, WANG L, XU J, et al. Revisiting local descriptor based image—to—class measure for few—shot learning [C]//Proceedings of the IEEE/CVF Conference on Com—puter Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019:7260-7268.

[27] ASKARI F, FATEH A, MOHAMMADI M R. Enhancing few—shot image classification through learnable multi—scale embedding and attention mechanisms[J].Neural Networks, 2025, 187:107339.

[28] CHEN Y, XU Z, WANG J, et al. Adaptive set—level metric for few—Shot image classification[J].Neural Networks, 2025:107924.

[29] DONG M, LI F, LI Z, et al. PRSN: Prototype resynthesis network with cross—image semantic alignment for few—shot image classification[J].Pattern Recognition, 2025, 159:111122

[30] DONG X, ZHU L, ZHANG D, et al. Fast parameter adapta—tion for few—shot image captioning and visual question answering[C]//Proceedings of the 26th ACM International Conference on Multimedia. Seoul, Republic of Korea: ACM, 2018:54-62.

[31] ZHANG J, LIU X, CHEN M, et al. Cross—modal feature distribution calibration for few—shot visual question answer—ing[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver, Canada: AAAI Press, 2024, 38(7):7151-7159.

[32] YIN C, WU K, CHE Z, et al. Hierarchical graph attention network for few—shot visual—semantic learning[C]//Proceed—ings of the IEEE/CVF International Conference on Com—puter Vision. Montreal, Canada: IEEE, 2021:2177-2186.

[33] REN M, KIROS R, ZEMEL R. Exploring models and data for image question answering[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS 2015). Montreal, Canada: MIT Press, 2015:2953-2961.

[34] VINYALS O, BLUNDELL C, LILLICRAP T, et al. Matching networks for one shot learning[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS 2016). Barcelona, Spain: Curran Associates Inc, 2016:3637-3645.

[35] SUNG F, YANG Y, ZHANG L, et al. Learning to compare: Relation network for few—shot learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018:1199-1208.

[36] KIM J, KIM T, KIM S, et al. Edge—labeling graph neural network for few—shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019:11-20.

[37] GARCIA V, BRUNA J. Few—shot learning with graph neur—al networks [EB/OL]. (2017—11—10). https://arxiv.org/abs/1711.04043