基于退化感知的盲图像超分辨率网络

Blind Image Super-Resolution Network Based on Degradation-Aware

摘要

近年来，基于深度学习的超分辨率取得了显著进展，该类方法性能依赖于精准的训练数据对. 现有方法大多采用双三次下采样获得低分辨率图像用于训练，即仅基于由固定且已知的退化方法产生的低分辨率图像展开. 然而，该类方法在处理具有复杂且未知退化的图像时，超分辨率性能急剧下降. 因此，部分研究针对具备复杂未知退化的图像盲超分辨率问题展开探索，并通过退化估计得到的先验信息进行超分辨率，但是这些方法缺乏对于退化特异性先验信息的利用. 针对此问题，本文提出退化感知的盲图像超分辨率网络. 首先，基于对比学习方法，设计了与图像内容解耦的退化表征学习模块，用以学习退化未知的图像的退化表征. 通过针对内容与退化差异构建不同类型的负样本，该模块提高了对于退化特异性信息的表征能力. 然后，在退化调制的多头自注意力模块基础上，提出了退化类别约束的多头自注意力模块，以利用得到的退化表征指导后续的超分辨率重建过程. 所提退化表征加强了特征提取过程中对结构和高频信息的捕捉能力，使得网络进一步根据图像退化信息自适应地进行重建. 实验结果表明，相较于其他先进的图像超分辨率方法，该网络在具有不同退化的测试图像上均获得了更优越的超分辨率重建性能.

Abstract

In recent years，significant progress has been made in deep-learning-based super-resolution methods，whose performance relies on accurate training data pairs. Most existing methods use bicubic downsampling to generate low-resolution images for training，relying solely on low-resolution images with fixed and known degradation. However，these methods do not perform well on images with complex and unknown degradation. Therefore，some studies explore blind super-resolution for images with complex and unknown degradation and used prior information obtained from the degradation estimation for super-resolution. However，these methods do not use degradation-specific prior information. Thus，a degradation-aware blind super-resolution network is proposed. First，based on contrastive learning，a degradation representation learning module is designed，which can decouple from the image content to learn the degradation representation of images with unknown degradation. Specifically，negative samples are constructed on the basis of differences in content and degradation to enhance the discriminability of the degradation representation. The degradation representation is then used to guide the subsequent low-resolution reconstruction process. Based on the degradation-modulated multi-head self-attention module，a degradation-category-constrained multi-head attention module is proposed. The degradation representations enhance the network’s ability to capture structural and high-frequency information during feature extraction and enable the network to adaptively extract and reconstruct features based on the degradation information. The experimental results demonstrate that the proposed method outperforms other advanced super-resolution methods on test images with various degradation.

关键词

盲超分辨率 / 退化估计 / 多头自注意力

Key words

blind super-resolution / degradation estimation / multi-head self-attention(MSA)

参考文献

[1] Dong C, Loy C C, He K M, et al. Image super—resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2):295-307.

[2] Kim J, Lee J K, Lee K M. Deeply—recursive convolutional network for image super—resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas，USA， 2016:1637-1645.

[3] Ledig C, Theis L, Huszár F, et al. Photo—realistic single image super—resolution using a generative adversarial network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，USA， 2017:4681-4690.

[4] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas，USA， 2016:770-778.

[5] Kim J, Kwon L J, Mu L K. Accurate image super—resolution using very deep convolutional networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas，USA， 2016:1646-1654.

[6] Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super—resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop. Venice，Italy， 2017:136-144.

[7] Zhang Y L, Li K P, Li K, et al. Image super—resolution using very deep residual channel attention networks[C]// Proceedings of the European Conference on Computer Vision(ECCV). Munich，Germany， 2018:286-301.

[8] 李素梅, 马力, 石永莲. 梯度指导的快速轻型超分辨率重建密集残差网络[J]. 天津大学学报(自然科学与工程技术版), 2021, 54(4):346-355.

[9] Li Sumei, Ma Li, Shi Yonglian. Gradient—guided fast and lightweight image super—resolution with dense residual network[J]. Journal of Tianjin University(Science and Technology), 2021, 54(4):346-355(in Chinese).

[10] 孙旭, 李晓光, 李嘉锋, 等. 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43(5):697-709.

[11] Sun Xu, Li Xiaoguang, Li Jiafeng, et al. Review on deep learning based image super—resolution restoration algorithms[J]. Acta Automatica Sinica, 2017, 43(5):697-709(in Chinese).

[12] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words：Transformers for image recognition at scale[EB/OL]. https://arxiv.org/abs/2010.11929， 2020—10—22.

[13] Liang J Y, Cao J Z, Sun G L, et al. Swinir：Image restoration using swin transformer[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal，Canada， 2021:1833-1844.

[14] Liu Z, Lin Y T, Cao Y, et al. Swin transformer：Hierarchical vision transformer using shifted windows[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal，Canada， 2021:10012-10022.

[15] Chen X Y, Wang X T, Zhou J T, et al. Activating more pixels in image super—resolution transformer[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Vancouver，Canada， 2023:22367-22377.

[16] Bell—Kligler S, Shocher A, Irani M. Blind super—resolution kernel estimation using an internal—GAN[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver，Canada， 2019:284-293.

[17] Liang J Y, Zhang K, Gu S H, et al. Flow—based kernel prior with application to blind super—resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville，USA， 2021:10601-10610.

[18] Tao G P, Ji X Z, Wang W Z, et al. Spectrum—to—kernel translation for accurate blind image super—resolution[J]. Advances in Neural Information Processing Systems, 2021, 34:22643-22654.

[19] Gu J J, Lu H N, Zuo W M, et al. Blind super—resolution with iterative kernel correction[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach，USA， 2019:1604-1613.

[20] Luo Z W, Huang H B, Yu L, et al. Deep constrained least squares for blind image super—resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans，USA， 2022:17642-17652.

[21] Wang L G, Wang Y Q, Dong X Y, et al. Unsupervised degradation representation learning for blind super—resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville，USA， 2021:10581-10590.

[22] Zhang K, Liang J Y, van Gool L, et al. Designing a practical degradation model for deep blind image super—resolution[C]// Proceedings of the IEEE International Conference on Computer Vision. Montreal，Canada， 2021:4791-4800.

[23] Wang X T, Xie L B, Dong C, et al. Real—ESRGAN：Training real—world blind super—resolution with pure synthetic data[C]// Proceedings of the IEEE International Conference on Computer Vision. Montreal，Canada， 2021:1905-1914.

[24] Liu Q G, Gao P, Han K, et al. Degradation—aware self—attention based transformer for blind image super—resolution[J]. IEEE Transactions on Multimedia, 2024, 26:7516-7528.

[25] He K M, Fan H Q, Wu Y X, et al. Momentum contrast for unsupervised visual representation learning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，USA， 2020:9729-9738.

[26] Zhang K, Zuo W M, Chen Y J, et al. Beyond a Gaussian denoiser：Residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing, 2017, 26(7):3142-3155.

基金资助

国家自然科学基金资助项目(62472308)