频率空间协作的红外可见光图像融合网络

Frequency-spatial Collaboration Network for Infrared and Visible Image Fusion

摘要

红外可见光图像融合旨在生成既突出显著目标又包含丰富纹理的图像.现有的融合方法主要关注空间域特征,忽略了频率域信息.因此,本文提出一种频率空间协作的红外可见光图像融合网络.首先通过频率分解模块将源图像分解为高频部分(模态特有特征)和低频部分(模态共有特征).同时,粗略提取源图像的空间域特征,以保留良好的空间结构.最后,利用跨域自适应融合模块学习自适应权重,动态调整频率域和空间域特征,以缓解域间差异并生成高质量融合图像.在TNO和VOT2020-RGBT数据集上的定量和定性实验结果表明,本文方法在6项评价指标上表现优异,且相比7种先进的融合方法,能更有效地融合多模态互补信息,生成显著性目标突出、细节丰富的融合图像.

Abstract

Infrared and visible image fusion aims to generate a single image highlighting salient objects and rich textures.Existing fusion methods predominantly focus on spatial characteristics while ignoring valuable frequency information.Therefore,this paper proposes a frequency-spatial collaboration network for infrared and visible image fusion.Firstly,the frequency decomposition module decomposes the source image into high-frequency(modality-specific features)and low-frequency components(modality-shared features).Simultaneously,the spatial characteristics are roughly extracted to preserve the spatial structure of the fused image.Finally,the cross-domain adaptive fusion module learns adaptive weights to dynamically adjust features in both frequency and spatial domains,thereby mitigating inter-domain differences and generating high-quality fused images.Quantitative and qualitative experimental results on the TNO and VOT2020-RGBT datasets demonstrate that the proposed method performs excellently across six evaluation metrics.Compared with seven state-of-the-art methods,it effectively integrates multi-modal complementary information and generates fused images with prominent salient targets and fine-grained texture information.

关键词

红外图像 / 可见光图像 / 图像融合 / 频率分解 / 跨域自适应融合

Key words

infrared image / visible image / image fusion / frequency decomposition / cross-domain adaptive fusion

[1] Liu J,Fan X,Huang Z,et al.Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2022:5792-5801.
[2] YU M Y,LIU X Y.Ship detection method based on multimodal visible and infrared image fusion[J/OL].Computer Engineering,2025,1-10,https://doi.org/10.19678/j.issn.1000-3428.0070436,2025-01-15.
[3] WANG Y T,LIU Z M,WAN Y P,et al.Target detection under low light conditions based on visible and infrared images[J].Computer Engineering,2024,50(8):270-281.
[4] CHENG Q H,JIAN H F,ZHENG S K,et al.Illumination-aware infrared/visible fusion for object detection[J].Computer Science,2025,52(2):173-182.
[5] Tang Z,Xu T,Li H,et al.Exploring fusion strategies for accurate RGBT visual object tracking[J].Information Fusion,2023,99:101-881,doi:10.48550/arXiv.2201.08673.
[6] Wang D,Liu J,Liu R,et al.An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection[J].Information Fusion,2023,98:101-828,doi:10.1016/j.inffus.2023.101828.
[7] LIU D,ZHANG G Y,SHI Y Q,et al.Zero-shot infrared and visible image fusion based on fusion curve[J].Pattern Recognition and Artificial Intelligence,2025,38(3):268-279.
[8] Li H,Wu X J.DenseFuse:a fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing,2019,28(5):2614-2623.
[9] YANG Y,LIU J X,HUANG S Y,et al.Convolutional auto-encoding fusion network for infrared and visible image fusion[J].Journal of Chinese Computer Systems,2019,40(12):2673-2680.
[10] QIU D F,HU X Y,LIANG P W,et al.A deep progressive infrared and visible image fusion network[J].Journal of Image and Graphics,2023,28(1):156-165.
[11] Tang L,Yuan J,Ma J.Image fusion in the loop of high-level vision tasks:a semantic-aware real-time infrared and visible image fusion network[J].Information Fusion,2022,82:28-42,doi:10.1016/j.inffus.2021.12.004.
[12] Li H,Xu T,Wu X,et al.LRRNet:a novel representation learning guided fusion network for infrared and visible images[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(9):11040-11052.
[13] Ma J,Yu W,Liang P,et al.FusionGAN:a generative adversarial network for infrared and visible image fusion[J].Information Fusion,2019,48:11-26,doi:10.1016/j.inffus.2018.09.004.
[14] Zhou H,Wu W,Zhang Y,et al.Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network[J].IEEE Transactions on Multimedia,2021,25:635-648,doi:10.1109/TMM.2021.3129609.
[15] Tang W,He F,Liu Y.YDTR:infrared and visible image fusion via Y-shape dynamic transformer[J].IEEE Transactions on Multimedia,2022,25:5413-5428,doi:10.1109/TMM.2022.3192661.
[16] SUN X H,GUAN Z,WANG X.Vision transformer for fusing infrared and visible images in groups[J].Journal of Image and Graphics,2023,28(1):166-178.
[17] Chen Y,Fan H,Xu B,et al.Drop an octave:reducing spatial redundancy in convolutional neural networks with octave convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,2019:3434-3443.
[18] Tang L,Yuan J,Zhang H,et al.PIAFusion:a progressive infrared and visible image fusion network based on illumination aware[J].Information Fusion,2022,83-84:79-92,doi:10.1016/j.inffus.2022.03.007.
[19] Li H,Wu X.CrossFuse:a novel cross attention mechanism based infrared and visible image fusion approach[J].Information Fusion,2024,103:102147,doi:10.1016/j.inffus.2023.102147.
[20] Alexander T.TNO image fusion dataset[J].Figshare Dataset,2014,doi:10.6084/m9.figshare.1008029.v2.
[21] Kristan M,Leonardis A,Matas J,et al.The eighth visual object tracking VOT2020 challenge results[C]//Computer Vision-ECCV Workshops,European Conferenceon Computer Vision,2020:1-56.
[22] Liu J,Fan X,Jiang J,et al.Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion[J].IEEE Transactions on Circuits and Systems for Video Technology,2022,32(1):105-119.
[23] Liang P,Jiang J,Liu X,et al.Fusion from decomposition:a self-supervised decomposition approach for image fusion[C]//European Conference on Computer Vision,2022,doi:10.1007/978-3-031-19797-0_41.
[24] Zhao Z,Bai H,Zhu Y,et al.DDFM:denoising diffusion model for multi-modality image fusion[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,2023:8082-8093.
[25] Wang X,Guan Z,Qian W,et al.CS2Fusion:contrastive learning for self-supervised infrared and visible image fusion by estimating feature compensation map[J].Information Fusion,2024,102:1-15,doi:10.1016/j.inffus.2023.102039.
[26] Roberts J W,Van Aardt J A,Ahmed F B.Assessment of image fusion procedures using entropy,image quality,and multispectral classification[J].Journal of Applied Remote Sensing,2008,2(1):1-28.
[27] MA Jiayi,MA Yong,LI Chang.Infrared and visible image fusion methods and applications:a survey[J].Information Fusion,2019,45:153-178,doi:10.1016/j.inffus.2018.02.004.
[28] Wang Z,Bovik A C,Sheikh H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[29] Yu H,Cai Y,Cao Y,et al.A new image fusion performance metric based on visual information fidelity[J].Information Fusion,2013,14(2):127-135.
[30] Cui G,Feng H,Xu Z,et al.Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J].Optics Communications,2015,341:199-209,doi:10.1016/j.optcom.2014.12.032.
[31] Haghighat M,Razian M A.Fast-FMI:non-reference image fusion metric[C]//IEEE 8th International Conference on Application of Information and Communication Technologies,2014:1-3.

附中文参考文献:
[2] 于梦源,刘向阳.基于多模态可见光和红外图像融合的船舶检测方法[J/OL].计算机工程,2025,1-10,https://doi.org/10.19678/j.issn.1000-3428.0070436,2025-01-15.
[3] 王昱婷,刘志明,万亚平,等.基于可见光与红外图像的弱光条件下目标检测[J].计算机工程,2024,50(8):270-281.
[4] 程清华,鉴海防,郑帅康,等.基于光照感知的红外/可见光融合目标检测[J].计算机科学,2025,52(2):173-182.
[7] 刘 铎,张国印,史一岐,等.基于融合曲线的零样本红外与可见光图像融合方法[J].模式识别与人工智能,2025,38(3):268-279.
[9] 杨 勇,刘家祥,黄淑英,等.卷积自编码融合网络的红外可见光图像融合[J].小型微型计算机系统,2019,40(12):2673-2680.
[10] 邱德粉,胡星宇,梁鹏伟,等.红外与可见光图像渐进融合深度网络[J].中国图象图形学报,2023,28(1):156-165.
[16] 孙旭辉,官 铮,王 学.红外与可见光图像分组融合的视觉Transformer[J].中国图象图形学报,2023,28(1):166-178.

基金资助

国家自然科学基金项目(U21A20487)资助.