基于跨模态生成与交叉注意力的乳腺超声诊断

Breast ultrasound diagnosis based on cross-modal generation and cross-attention

摘要

目的：针对常规B超在乳腺癌筛查中因读图者间一致性不佳、微小病灶恶性征象不典型导致的准确性受限问题，以及超声造影因成本高、操作复杂而难以普及的临床矛盾，提出一种“分割-生成-分类”三阶段级联诊断框架，旨在仅利用常规B超图像实现高精度的乳腺肿瘤良恶性诊断。方法：首先，采用CNN-Transformer架构的分割网络从B超图像中提取肿瘤区域。接着，构建基于条件生成对抗网络的跨模态生成模型，利用预训练的ResNet-34编码器从分割后的B超中生成对应的超声造影图像。最后，设计以Inception-v3为骨干的双分支交叉注意力网络，通过动态特征融合实现对真实B超与生成造影信息的深度交互，完成对肿瘤良恶性的精确诊断。结果：在包含677例患者的乳腺超声数据集上进行验证，该框架的受试者工作特征曲线下面积（AUC）达到0.930，显著优于多个单模态B超基线模型（最佳AUC为0.868）。消融实验证实病灶分割、跨模态生成以及交叉注意力特征融合模块对提升诊断性能的必要贡献。结论：本研究提出的框架验证了通过深度学习方法生成并利用多模态超声信息进行高精度诊断的可行性与有效性，能够以低成本、高效率的方式提供多模态诊断增益，有望为乳腺癌临床筛查提供有力的辅助决策支持，降低乳腺癌临床筛查的漏诊与误诊风险。

Abstract

Objective To address the limited accuracy of conventional B-mode ultrasound (US) in breast cancer screening, resulting from poor inter-reader agreement and atypical malignant signs of tiny lesions, and resolve the clinical dilemma where contrast-enhanced ultrasound (CEUS) is hard to popularize due to its high cost and operational complexity, this study proposes a three-stage cascaded diagnostic framework of "segmentation-generation-classification" to achieve high-precision diagnosis of benign and malignant breast tumors relying solely on conventional US images. Methods A CNN-Transformer framework-based segmentation network was first employed to extract tumor regions from US images. Subsequently, a cross-modal generation model based on a conditional generative adversarial network was constructed, utilizing a pre-trained ResNet-34 encoder to synthesize corresponding CEUS images from the segmented US images. Finally, a dual-branch cross-attention network with Inception-v3 as the backbone was designed to achieve deep interaction between the real US and the generated CEUS information through dynamic feature fusion, thereby accomplishing accurate benign and malignant tumor classification. Results Validated on a breast ultrasound dataset including 677 patients, the proposed framework achieved an area under the receiver operating characteristic curve (AUC) of 0.930, significantly outperforming several single-modal US baseline models which reached a maximum AUC of 0.868. Ablation experiments verified the essential contributions of lesion segmentation, cross-modal generation, and cross-attention feature fusion modules in enhancing diagnostic performance. Conclusion This study validates the feasibility and effectiveness of generating and utilizing multimodal ultrasound information with deep learning for high-precision diagnosis. The proposed framework provides multimodal diagnostic benefits in a cost-effective and highly efficient way, holding great promise to provide robust auxiliary decision support for clinical breast cancer screening and reduce the risks of missed diagnoses and misdiagnoses.

关键词

乳腺癌 / 乳腺超声 / 超声造影 / 辅助诊断 / 生成对抗网络 / 跨模态生成 / 交叉注意力 / 特征融合

Key words

breast cancer / breast ultrasound / contrast-enhanced ultrasound / auxiliary diagnosis / generative adversarial network / cross-modal generation / cross-attention / feature fusion

参考文献

[1] Bray F, Laversanne M, Sung H, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries[J]. CA Cancer J Clin, 2024, 74(3): 229-263.

[2] Sung H, Ferlay J, Siegel RL, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries[J]. CA Cancer J Clin, 2021, 71(3): 209-249.

[3] Hooley RJ, Scoutt LM, Philpotts LE . Breast ultrasonography: state of the art[J]. Radiology, 2013, 268(3): 642-659.

[4] Berg WA, Blume JD, Cormack JB, et al. Combined screening with ultrasound and mammography vs mammography alone in women at elevated risk of breast cancer [J]. JAMA, 2008, 299(18): 2151-2163.

[5] Abdullah N, Mesurolle B, El—Khoury M, et al. Breast imaging reporting and data system lexicon for US: interobserver agreement for assessment of breast masses[J]. Radiology, 2009, 252(3): 665-672.

[6] Romeo V, Cuocolo R, Apolito R, et al. Clinical value of radiomics and machine learning in breast ultrasound: a multicenter study for differential diagnosis of benign and malignant lesions[J]. Eur Radiol, 2021, 31(12): 9511-9519.

[7] Chen C, Turco S, Kapetas P, et al. Spatiotemporal analysis of contrast—enhanced ultrasound for differentiating between malignant and benign breast lesions[J]. Eur Radiol, 2024, 34(7): 4764-4773.

[8] Li Q, Hu M, Chen ZK, et al. Meta—analysis: contrast—enhanced ultrasound versus conventional ultrasound for differentiation of benign and malignant breast lesions[J]. Ultrasound Med Biol, 2018, 44(5): 919-929.

[9] Cheng M, Tong W, Luo J, et al. Value of contrast—enhanced ultrasound in the diagnosis of breast US—BI—RADS 3 and 4 lesions with calcifications[J]. Clin Radiol, 2020, 75(12): 934-941.

[10] Wan CF, Du J, Fang H, et al. Evaluation of breast lesions by contrast enhanced ultrasound: qualitative and quantitative analysis[J]. Eur J Radiol, 2012, 81(4): e444-e450.

[11] Sidhu PS, Cantisani V, Dietrich CF, et al. The EFSUMB guidelines and recommendations for the clinical practice of contrast—enhanced ultrasound (CEUS) in non—hepatic applications: update 2017 (long version)[J]. Ultraschall Med, 2018, 39(2): e2-e44.

[12] Wang J, Zhao R, Cheng J . Diagnostic accuracy of contrast—enhanced ultrasound to differentiate benign and malignant breast lesions: a systematic review and meta—analysis[J]. Eur J Radiol, 2022, 149: 110219.

[13] 孙雨, 杨琛 . 基于自动乳腺超声诊断系统的乳腺肿瘤人工智能诊断研究进展[J]. 中国医学影像学杂志, 2024, 32(11): 1176-1181.

[14] Sun Y, Yang C . Reasearch progress on artificial intelligence diagnosis for breast tumors based on automated breast ultrasound system[J]. Chinese Journal of Medical Imaging, 2024, 32(11): 1176-1181.

[15] Ronneberger O, Fischer P, Brox T . U—Net: convolutional networks for biomedical image segmentation[C]// Medical Image Computing and Computer—Assisted Intervention—MICCAI 2015. Cham: Springer International Publishing, 2015: 234-241.

[16] 王一凡, 刘静, 马金刚, 等 . 深度学习在乳腺癌影像学检查中的应用进展[J]. 计算机科学与探索, 2024, 18(2): 301-319.

[17] Wang YF, Liu J, Ma JG, et al. Application progress of deep learning in imaging examination of breast cancer[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(2): 301-319.

[18] Cao H, Wang YY, Chen J, et al. Swin—Unet: Unet—like pure transformer for medical image segmentation[C]// Computer Vision—ECCV 2022 Workshops. Cham: Springer Nature Switzerland, 2023: 205-218.

[19] Chen JN, Lu YY, Yu QH, et al. TransUNet: transformers make strong encoders for medical image segmentation[EB/OL]. (2021—02—08). https://arxiv.org/abs/2102.04306.

[20] 柴梦婷, 朱远平 . 生成式对抗网络研究与应用进展[J]. 计算机工程, 2019, 45(9): 222-234.

[21] Chai MT, Zhu YP . Research and application progress of generative adversarial networks[J]. Computer Engineering, 2019, 45(9): 222-234.

[22] Goodfellow I, Pouget—Abadie J, Mirza M, et al. Generative adversarial networks[J]. Commun ACM, 2020, 63(11): 139-144.

[23] Isola P, Zhu JY, Zhou TH, et al. Image—to—image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ, USA: IEEE, 2017: 5967-5976.

[24] Nie D, Trullo R, Lian J, et al. Medical image synthesis with context—aware generative adversarial networks[C]// Medical Image Computing and Computer Assisted Intervention—MICCAI 2017. Cham: Springer International Publishing, 2017: 417-425.

[25] Chen C, Yu JB, Xu ZK, et al. Pseudo—contrast—enhanced US via enhanced generative adversarial networks for evaluating tumor ablation efficacy [J]. Radiol Artif Intell, 2025, 7(3): e240370.

[26] Wang QC, Chen H, Luo GN, et al. Performance of novel deep learning network with the incorporation of the automatic segmentation network for diagnosis of breast cancer in automated breast ultrasound[J]. Eur Radiol, 2022, 32(10): 7163-7172.

[27] Zhou HY, Yu YZ, Wang CD, et al. A transformer—based representation—learning model with unified processing of multimodal input for clinical diagnostics[J]. Nat Biomed Eng, 2023, 7(6): 743-755.

[28] Chen C, Wang Y, Niu JW, et al. Domain knowledge powered deep learning for breast cancer diagnosis based on contrast—enhanced ultrasound videos[J]. IEEE Trans Med Imaging, 2021, 40(9): 2439-2451.

[29] 何俊, 张彩庆, 李小珍, 等 . 面向深度学习的多模态融合技术研究综述[J]. 计算机工程, 2020, 46(5): 1-11.

[30] He J, Zhang CQ, Li XZ, et al. Survey of research on multimodal fusion technology for deep learning[J]. Computer Engineering, 2020, 46(5): 1-11.

[31] Chen CFR, Fan QF, Panda R . CrossViT: cross—attention multi—scale vision transformer for image classification[C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Piscataway, NJ, USA: IEEE, 2021: 347-356.

[32] Ab Mumin N, Ramli Hamid MT, Wong JHD, et al. Magnetic resonance imaging phenotypes of breast cancer molecular subtypes: a systematic review[J]. Acad Radiol, 2022, 29 Suppl 1: S89-S106.

[33] Zhang K, Li JW, Zhu Q, et al. Prediction of pathologic complete response by ultrasonography and magnetic resonance imaging after neoadjuvant chemotherapy in patients with breast cancer[J]. Cancer Manag Res, 2020, 12: 2603-2612.

[34] Gong XT, Li QF, Gu LS, et al. Conventional ultrasound and contrast—enhanced ultrasound radiomics in breast cancer and molecular subtype diagnosis[J]. Front Oncol, 2023, 13: 1158736.

[35] Ko ES, Lee BH, Kim HA, et al. Triple—negative breast cancer: correlation between imaging and pathological findings[J]. Eur Radiol, 2010, 20(5): 1111-1117.

基金资助

国家自然科学基金(52175020)

上海市自然科学基金(21ZR1451400)