基于多重分形优化的图像超分辨率重建

Image super-resolution reconstruction based on multifractal optimization

摘要

图像超分辨率旨在从低分辨率图像中重建包含丰富纹理细节的高分辨率图像. 已有研究表明,精细结构主要对应于傅里叶域中的高频成分,但多数现有方法缺乏针对高频信息的自适应处理机制,容易导致边缘模糊或纹理紊乱. 针对上述问题,本文在非下采样轮廓波变换(non-subsampled contourlet transform, NSCT)域内开展超分辨率研究,将图像分解为多个具有不同频率特性的子带. 不同子带所包含的频率信息与图像细节程度密切相关,本文将其定义为图像粗糙度特征. 此外,对各子带图像应用分形分析方法,利用分形对图像粗糙度的数学表征能力,描述图像信息的多种频率细节. 在此基础上,构建多子带下的多重分形模型,并在各子带上自适应生成相应的分形表示,从而将超分辨率重建过程转化为一个多重分形优化问题. 实验结果表明,本研究所提出的方法能够有效恢复图像高频细节.

Abstract

The goal of image super-resolution technique is to reconstruct high-resolution image with fine details and vivid texture details from its low-resolution version. On Fourier domain, such fine details are more related to the information in the high-frequency spectrum. Most of existing methods do not have specific modules to handle such high-frequency information adaptively. Thus, they cause edge blur or texture disorder. To tackle the problems, this work explores image super-resolution on multiple sub-bands of the corresponding image, which are generated by non-subsampled contourlet transform (NSCT). Different sub-band hold the information of different frequency which is then related to the detailedness of information of the given low-resolution image. In this work, such image information detailedness is formulated as image roughness. Moreover, fractals analysis is applied to every single sub-band image. Since fractals can mathematically represent the image roughness, it then is able to represent the detailedness (i.e. various frequency of image information). Overall, a multi-fractals formulation is established based on multiple sub-bands image. On each sub-band image, different fractals representation is created adaptively according to image features. In this way, the image super-resolution process is transformed into a multifractal optimization problem. The experiment result demonstrates the effectiveness of the proposed method in recovering high-frequency details.

关键词

图像超分辨 / 非下采样轮廓波变换 / 分形插值函数 / 图像粗糙度 / 分形维度 / 多重分形

Key words

image super-resolution / nonsubsampled contourlet transform / fractal interpolation function / image roughness / fractal dimension / multifractal

参考文献

[1] ZHOU F Q, LI X J, LI Z X. High—frequency detail enhancing DenseNet for super—resolution[J]. Neurocomputing, 2018, 290: 34-42.

[2] LIU Y, WANG Y C, LIN, et al. An attention—based approach for single image super resolution[C]// 2018 24th International Conference on Pattern Recognition (ICPR). Beijing: IEEE, 2018: 2777-2784.

[3] CUI Z, CHANG H, SHAN S G, et al. Deep network cascade for image super—resolution[C]// Computer Vision—ECCV 2014. Cham: Springer, 2014: 49-64.

[4] HUANG H B, HE R, SUN Z N, et al. Wavelet—SRNet: a wavelet—based CNN for multi—scale face super resolution[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, 2017: 1698-1706.

[5] GRIBBON K T, BAILEY D G. A novel approach to real—time bilinear interpolation[C]// Proceedings of DELTA 2004. Second IEEE International Workshop on Electronic Design, Test and Applications. Perth: IEEE, 2005: 126-131.

[6] ZHANG C M, ZHANG X, LI X M, et al. Cubic surface fitting to image with edges as constraints[C]// 2013 IEEE International Conference on Image Processing. Melbourne: IEEE, 2014: 1046-1050.

[7] MATSUMOTO S, KAMADA M, MIJIDDORJ R O. Adaptive image interpolation by cardinal splines in piecewise constant tension[J]. Optimization Letters, 2012, 6(7): 1265-1280.

[8] SUN J, SUN J, XU Z B, et al. Gradient profile prior and its applications in image super—resolution and enhancement[J]. IEEE Transactions on Image Processing, 2011, 20(6): 1529-1542.

[9] PICKUP L, ROBERTS S, ZISSERMAN A. A sampled texture prior for image super—resolution[J]. Advances in neural information processing systems, 2003, 16: 1-8.

[10] ZHAO J W, HU H P, CAO F L. Image super—resolution via adaptive sparse representation[J]. Knowledge—Based Systems, 2017, 124: 23-33.

[11] ZHANG H C, YANG J C, ZHANG Y N, et al. Image and video restorations via nonlocal kernel regression[J]. IEEE Transactions on Cybernetics, 2013, 43(3): 1035-1046.

[12] FATTAL R. Image upsampling via imposed edge statistics[C]// 34th Annual Meeting of the Association for Computing Machinery̓s Special Interest Group on Graphics. San Diego: ACM, 2007: 95.

[13] TAI Y W, LIU S C, BROWN M S, et al. Super resolution using edge prior and single image detail synthesis[C]// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Francisco: IEEE, 2010: 2400-2407.

[14] YAN Q, XU Y, YANG X K, et al. Single image superresolution based on gradient profile sharpness[J]. IEEE Transactions on Image Processing, 2015, 24(10): 3187-3202.

[15] BAKER S, KANADE T. Limits on super—resolution and how to break them[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(9): 1167-1183.

[16] LIU H, CAO F L, WEN C L, et al. Lightweight multi—scale residual networks with attention for image super—resolution[J]. Knowledge—Based Systems, 2020, 203: 106103.

[17] TIAN C W, ZHU G E, WU Z H, et al. Lightweight image super—resolution with enhanced CNN[J]. Knowledge—Based Systems, 2020, 205: 106235.

[18] DU N Y J, DA Z Y, YANG S, et al. Kernel—attended residual network for single image super—resolution[J]. Knowledge—Based Systems, 2021, 213: 106663.

[19] ESMAEILZEHIA, AHMAD M O, SWAMY M N S . MuRNet: a deep recursive network for super resolution of bicubically interpolated images[J]. Signal Processing: Image Communication, 2021, 94: 116228.

[20] ZARE A POOR M, CELEBI M E, YANG J. Diverse adversarial network for image super—resolution[J]. Signal Processing: Image Communication, 2019, 74: 191-200.

[21] WANG Q, TANG X O, SHUM H. Patch based blind image super resolution[C]// Tenth IEEE International Conference on Computer Vision (ICCV'05), Volume 1. Beijing: IEEE, 2005: 709-716.

[22] CHANG H, YEUNG D Y, XIONG Y M. Super—resolution through neighbor embedding[C]// Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004. Washington, DC: IEEE, 2004: 1.

[23] YANG J C, WRIGHT J, HUANG T S, et al. Image super—resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.

[24] ZHANG J, SHAO M H, YU L L, et al. Image super—resolution reconstruction based on sparse representation and deep learning[J]. Signal Processing: Image Communication, 2020, 87: 115925.

[25] ZHANG K B, GAO X B, TAO D C, et al. Single image super—resolution with non—local means and steering kernel regression[J]. IEEE Transactions on Image Processing, 2012, 21(11): 4544-4556.

[26] HE H, SIU W C. Single image super—resolution using Gaussian process regression[C]// CVPR 2011. Colorado Springs: IEEE, 2011: 449-456.

[27] TIAN Y P, ZHOU F, YANG W M, et al. Anchored neighborhood regression based single image super—resolution from self—examples[C]// 2016 IEEE International Conference on Image Processing (ICIP). Phoenix: IEEE, 2016: 2827-2831.

[28] WEI X Y, DRAGOTTI P L. FRESH: FRI—based single—image super—resolution algorithm[J]. IEEE Transactions on Image Processing, 2016, 25(8): 3723-3735.

[29] HUANG J B, SINGH A, AHUJA N. Single image super—resolution from transformed self—exemplars[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 5197-5206.

[30] HAN Z J, DENNEY T S. Interpolation of 2—D fractional Brownian motion using first order increments[C]// Proceedings 1998 International Conference on Image Processing. ICIP 98. Chicago: IEEE, 2002: 222-226.

[31] HAN Z J, DENNEY T S. Incremental Fourier interpolation of 2—D fractional Brownian motion[J]. IEEE Transactions on Industrial Electronics, 2001, 48(5): 920-925.

[32] WEE Y C, SHIN H J. A novel fast fractal super resolution technique[J]. IEEE Transactions on Consumer Electronics, 2010, 56(3): 1537-1541.

[33] XU H T, ZHAI G T, YANG X K. Single image super—resolution with detail enhancement based on local fractal analysis of gradient[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2013, 23(10): 1740-1754.

[34] ZHANG Y F, FAN Q L, BAO F X, et al. Single—image super—resolution based on rational fractal interpolation[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3782-3797.

[35] ZHANG Y F, WANG P, BAO F X, et al. A single—image super—resolution method based on progressive—iterative approximation[J]. IEEE Transactions on Multimedia, 2020, 22(6): 1407-1422.

[36] KRUPIN'SKI M, WAWRZASZEK A, DRZEWIECKI W, et al. What can multifractal analysis tell us about hyperspectral imagery?[J]. Remote Sensing, 2020, 12(24): 4077.

[37] 王晨泽, 沈雪豪, 黄振立, 等. 基于定位数据的全景超分辨图像交互可视化框架[J]. 计算机研究与发展, 2024, 61(7): 1741-1753.

[38] WANG Chenze, SHEN Xuehao, HUANG Zhenli, et al. Interactive visualization framework for panoramic super—resolution images based on localization data[J]. Journal of Computer Research and Development, 2024, 61(7): 1741-1753.

[39] 范清兰, 张云峰, 包芳勋, 等. 基于NSCT的区域自适应图像插值算法[J]. 计算机研究与发展, 2018, 55(3): 629-642.

[40] FAN Qinglan, ZHANG Yunfeng, BAO Fangxun, et al. An region adaptive image interpolation algorithm based on the NSCT[J]. Journal of Computer Research and Development, 2018, 55(3): 629-642.

[41] DU C J , SUN D W. Recent developments in the applications of image processing techniques for food quality evaluation[J]. Trends in Food Science & Technology, 2004, 15(5): 230-249.

[42] MENDOZA F, DEJMEK P, AGUILERA J M. Colour and image texture analysis in classification of commercial potato chips[J]. Food Research International, 2007, 40(9): 1146-1154.

[43] LEE K C, HO S J, HO S Y. Accurate estimation of surface roughness from texture features of the surface image using an adaptive neuro—fuzzy inference system[J]. Precision Engineering, 2005, 29(1): 95-100.

[44] YAO X X, WU Q, ZHANG P, et al. Weighted adaptive image super—resolution scheme based on local fractal feature and image roughness[J]. IEEE Transactions on Multimedia, 2021, 23: 1426-1441.

[45] CHEN Z Y, MURAMATSU S, ABE Y. Fast image super—resolution via multiple directional transforms[C]// 2016 IEEE International Conference on Image Processing (ICIP). Phoenix: IEEE, 2016: 1434-1438.

[46] FRITSCHE M, GU S, TIMOFTE R. Frequency separation for real—world super—resolution[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul: IEEE, 2019: 3599-3608.

[47] DA CUNHA A L, ZHOU J, DO M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10): 3089-3101.

[48] BARNSLEY M F. Fractal functions and interpolation[J]. Constructive Approximation, 1986, 2(1): 303-329.

[49] MANDELBROT B B. The fractal geometry of nature[M]. New York: W H Freeman and Company, 1982: 131-147.

[50] BOUDA M, CAPLAN J S, SAIERS J E. Box—counting dimension revisited: presenting an efficient method of minimizing quantization error and an assessment of the self—similarity of structural root systems[J]. Frontiers in Plant Science, 2016, 7: 149.

[51] BAO F X, YAO X X, SUN Q H, et al. Smooth fractal surfaces derived from bicubic rational fractal interpolation functions[J]. Science China Information Sciences, 2018, 61(9): 099104.

[52] EBERHART, SHI Y H. Particle swarm optimization: developments, applications and resources[C]// Proceedings of the 2001 Congress on Evolutionary Computation. Seoul: IEEE, 2002: 81-86.

[53] TIMOFTE R, DE V, VAN GOOL L. Anchored neighborhood regression for fast example—based super—resolution[C]// 2013 IEEE International Conference on Computer Vision. Sydney: IEEE, 2014: 1920-1927.

[54] DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super—resolution[C]// Computer Vision—ECCV 2014. Cham: Springer, 2014: 184-199.

[55] LAI W S, HUANG J B, AHUJA N, et al. Fast and accurate image super—resolution with deep Laplacian pyramid networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613.

[56] SHOCHER A, COHEN N, IRANI M. Zero—shot super—resolution using deep internal learning[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3118-3126.

[57] ZEYDE R, ELAD M, PROTTER M. On single image scale—up using sparse—representations[C]// Curves and Surfaces. Berlin, Heidelberg: Springer, 2012: 711-730.

[58] FRANZEN R. Kodak lossless true color image suite[DS/OL]. (2024—10—25)[2026—01—14]. https://doi.org/10.48550/arXiv.2402.16855.

基金资助

国家自然科学基金资助项目(62506209)

山东省自然科学基金项目(ZR2024QF016)

山东省自然科学基金项目(ZR2023QF161)

山东省高等学校青年创新团队项目(2022KJ185)