基于扩散的多模态医学图像融合

Diffusion-based multimodal medical image fusion

摘要

近年来，随着医学影像学技术的持续发展，图像融合技术在医学图像分析中得到了广泛应用。传统的融合技术受限于人工设计的特征提取过程，导致对图像语义信息的理解和匹配精度不高，且无法充分利用多模态图像的信息。通过研究一种基于扩散模型的多模态图像融合方法，利用扩散模型逐步学习多通道图像在潜在空间的联合特征，克服单一端到端网络学习能力有限的问题，生成高质量的融合图像，并对逆向去噪过程针对多模态医学图像融合任务做出了改进。引入了两个模态判别器加强去噪网络对于模态特征的理解，充分利用不同成像模态之间的互补性信息。在 AANLIB 数据集上的实验表明，该方法实现了令人满意的融合结果。

Abstract

In recent years, with the continuous development of medical imaging technology, image fusion techniques have been widely applied in medical image analysis. Traditional fusion methods are limited by manually designed feature extraction processes, resulting in limited accuracy in understanding and matching of image semantic information, and inability to fully utilize the information from multimodal images. A diffusion-based multimodal image fusion method is investigated in this paper. This method progressively learns the joint features of multi-channel images in the latent space using a diffusion model to overcome the limited learning capability of single end-to-end networks. And it generates high-quality fused images and improve the reverse denoising process specifically for the task of multimodal medical image fusion. Two modal discriminators are incorporated to enhance the denoising network’s understanding of modality-specific features, fully leveraging the complementary information between different imaging modalities. Experiments on the AANLIB dataset demonstrate that the proposed method achieves satisfactory fusion results.

关键词

深度学习 / 图像融合 / 扩散模型 / 多模态

Key words

deep learning / image fusion / diffusion model / multimodal

参考文献

[1] ABBASI S, TAVAKOLI M, BOVEIRI H R, et al. Medical image registration using unsupervised deep neural network: A scoping literature review[J]. Biomedical Signal Processing and Control, 2022, 73: 103444.

[2] 刘思捷 . 智慧医疗: 医学图像融合领域技术综述[J]. 专利代理, 2023(1): 39-45.

[3] LIU S J. Intelligent medical treatment: A review of medical image fusion technology[J]. Patent Agency, 2023(1): 39-45.

[4] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.

[5] 石争浩, 仵晨伟, 李成建, 等. 航空遥感图像深度学习目标检测技术研究进展[J]. 中国图象图形学报, 2023, 28(9): 2616-2643.

[6] SHI Z H, WU C W, LI C J, et al. Object detection techniques based on deep learning for aerial remote sensing images: A survey[J]. Journal of Image and Graphics, 2023, 28(9): 2616-2643.

[7] 罗福亚 . 基于视觉认知机理的夜间热红外图像彩色化方法研究[D]. 成都: 电子科技大学, 2023.

[8] LUO F Y. Research on colorization method of nighttime thermal infrared images based on visual cognitive mechanism[D]. Chengdu: University of Electronic Science and Technology of China, 2023.

[9] 王丽芳, 米嘉, 秦品乐, 等. 改进 U—Net3+与跨模态注意力块的医学图像融合[J]. 中国图象图形学报, 2022, 27(12): 3622-3636.

[10] WANG L F, MI J, QIN P L, et al. Medical image fusion using improved U—Net3+ and cross—modal attention blocks[J]. Journal of Image and Graphics, 2022, 27(12): 3622-3636.

[11] HERMESSI H, MOURALI O, ZAGROUBA E. Multimodal medical image fusion review: Theoretical background and recent advances[J]. Signal Processing, 2021, 183: 108036.

[12] YANG B, JING Z L, ZHAO H T. Review of pixel—level image fusion[J]. Journal of Shanghai Jiaotong University (Science), 2010, 15(1): 6-12.

[13] DU J, LI W S, LU K, et al. An overview of multi—modal medical image fusion[J]. Neurocomputing, 2016, 215: 3-20.

[14] SINGH S, ANAND R S. Ripplet domain fusion approach for CT and MR medical image information[J]. Biomedical Signal Processing and Control, 2018, 46: 281-292.

[15] SHABANZADE F, GHASSEMIAN H. Combination of wavelet and contourlet transforms for PET and MRI image fusion[C]// Proceedings of the Artificial Intelligence and Signal Processing Conference. New York: IEEE, 2017: 178-183.

[16] JIN Z Y, WANG Y J, CHEN Z G, et al. Medical image fusion in gradient domain with structure tensor[J]. Journal of Medical Imaging and Health Informatics, 2016, 6(5): 1314-1318.

[17] BHAVANA V, KRISHNAPPA H K. A survey on multi—modality medical image fusion[C]// Proceedings of the International Conference on Wireless Communications, Signal Processing and Networking. New York: IEEE, 2016: 1326-1329.

[18] EGFIN NIRMALA D, VAIDEHI V. Comparison of Pixel—level and feature level image fusion methods[C]// Proceedings of the 2nd International Conference on Computing for Sustainable Global Development. New York: IEEE, 2015: 743-748.

[19] XI X X, LUO X Q, ZHANG Z C, et al. Multimodal medical volumetric image fusion based on multi—feature in 3—D shearlet transform[C]// Proceedings of the International Smart Cities Conference. New York: IEEE, 2017: 1-6.

[20] TALBAR S N, CHAVAN S S, PAWAR A. Non—subsampled complex wavelet transform based medical image fusion[C]// Proceedings of the Future Technologies Conference (FTC) 2018. Cham: Springer International Publishing, 2018: 548-556.

[21] TANG L, LI L D, QIAN J S, et al. NSCT—based multimodal medical image fusion with sparse representation and pulse coupled neural network[J]. Journal of Information Hiding and Multimedia Signal Processing, 2016, 7(6): 1306-1316.

[22] MA J Y, MA Y, LI C. Infrared and visible image fusion methods and applications: A survey[J]. Information Fusion, 2019, 45: 153-178.

[23] VELMURUGAN S P, SIVAKUMAR P, RAJASEKARAN M P. Multimodality image fusion using centre—based genetic algorithm and fuzzy logic[J]. International Journal of Biomedical Engineering and Technology, 2018, 28(4): 322.

[24] SHAHDOOSTI H R, MEHRABI A. Multimodal image fusion using sparse representation classification in tetrolet domain[J]. Digital Signal Processing, 2018, 79: 9-22.

[25] 黄渝萍, 李伟生 . 医学图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1): 118-143.

[26] HUANG Y P, LI W S. A review of medical image fusion methods[J]. Journal of Image and Graphics, 2023, 28(1): 118-143.

[27] SONG J M, MENG C L, ERMON S. Denoising diffusion implicit models[EB/OL]. [ 2024—09—19]. https://arXiv.org/abs/2010.02502.

[28] ZHOU H B, WU W, ZHANG Y D, et al. Semantic—supervised infrared and visible image fusion via a dual—discriminator generative adversarial network[J]. IEEE Transactions on Multimedia, 2021, 25: 635-648.

[29] XU H, MA J Y, JIANG J J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.

[30] FU J, LI W S, DU J, et al. A multiscale residual pyramid attention network for medical image fusion[J]. Biomedical Signal Processing and Control, 2021, 66: 102488.

[31] YUE J, FANG L Y, XIA S B, et al. Dif—fusion: Toward high color fidelity in infrared and visible image fusion with diffusion models[J]. IEEE Transactions on Image Processing, 2023, 32: 5705-5720.

基金资助

国家自然科学基金(62076054)

国家自然科学基金(62027827)

国家自然科学基金(62072074)

国家自然科学基金(62372083)

四川省科技计划项目(2022JDJQ0039)