计算机科学技术学报 ›› 2020,Vol. 35 ›› Issue (3): 551-563.doi: 10.1007/s11390-020-0270-3

所属专题： Artificial Intelligence and Pattern Recognition； Computer Graphics and Multimedia

• Special Section of CVM 2020 • 上一篇    下一篇

基于渐进式通道注意力网络的参考图引导超分辨率研究

Huan-Jing Yue1, Member, IEEE, Sheng Shen1, Jing-Yu Yang1,*, Senior Member, IEEE, Hao-Feng Hu2, Yan-Fang Chen1   

  1. 1 School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China;
    2 School of Precision Instrument and Opto-Electronics Engineering, Tianjin University, Tianjin 300072, China
  • 收稿日期:2020-01-18 修回日期:2020-04-02 出版日期:2020-05-28 发布日期:2020-05-28
  • 通讯作者: Jing-Yu Yang E-mail:yjy@tju.edu.cn
  • 作者简介:Huan-Jing Yue received her B.S. and Ph.D. degrees in electronic and information engineering from Tianjin University, Tianjin, in 2010 and 2015, respectively. She was an intern with Microsoft Research Asia, Beijing, from 2011 to 2012, and from 2013 to 2015. She visited the Video Processing Laboratory, University of California at San Diego, from 2016 to 2017. She is currently an associate professor with the School of Electrical and Information Engineering, Tianjin University, Tianjin. Her current research interests include image processing and computer vision. She received the Microsoft Research Asia Fellowship Honor in 2013 and was selected into the Elite Scholar Program of Tianjin University in 2017.
  • 基金资助:
    This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61672378, 61771339, and 61520106002.

Reference Image Guided Super-Resolution via Progressive Channel Attention Networks

Huan-Jing Yue1, Member, IEEE, Sheng Shen1, Jing-Yu Yang1,*, Senior Member, IEEE, Hao-Feng Hu2, Yan-Fang Chen1        

  1. 1 School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China;
    2 School of Precision Instrument and Opto-Electronics Engineering, Tianjin University, Tianjin 300072, China
  • Received:2020-01-18 Revised:2020-04-02 Online:2020-05-28 Published:2020-05-28
  • Contact: Jing-Yu Yang E-mail:yjy@tju.edu.cn
  • About author:Huan-Jing Yue received her B.S. and Ph.D. degrees in electronic and information engineering from Tianjin University, Tianjin, in 2010 and 2015, respectively. She was an intern with Microsoft Research Asia, Beijing, from 2011 to 2012, and from 2013 to 2015. She visited the Video Processing Laboratory, University of California at San Diego, from 2016 to 2017. She is currently an associate professor with the School of Electrical and Information Engineering, Tianjin University, Tianjin. Her current research interests include image processing and computer vision. She received the Microsoft Research Asia Fellowship Honor in 2013 and was selected into the Elite Scholar Program of Tianjin University in 2017.
  • Supported by:
    This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61672378, 61771339, and 61520106002.

PDF (PC)

112

近年来，用于单张图像超分辨率重建(SISR)的卷积神经网络(CNN)变得越来越复杂，使得继续提升单图像超分辨率重建的性能变得更加具有挑战性。与此同时，基于参考图引导的图像超分辨率重建(RefSR)算法成为一种提高SR性能的有效策略。在RefSR中，引入的高分辨率(HR)参考图像可以提升高频残差的预测结果。然而现有的基于CNN的RefSR方法大都是简单地将低分辨率图像和高清参考图在通道维进行级联，平等地对待参考图(Ref)和低分辨率图(LR)中的有效信息。但是参考图和低分辨率输入图对最终SR结果的贡献不同,因此，本文提出了用于RefSR的渐进式通道注意力网络(PCANet)。本文有两个技术贡献。首先，本文提出了一种新颖的通道注意力模块(CAM)，该模块通过对空间特征进行加权平均而不是使用全局平均来估计通道加权参数。其次，考虑到当LR输入包含更多细节时可以改善残差预测过程，因此本文逐级执行超分辨率，这样可以利用多尺度的参考图像信息。图中展示的是本文所提出的超分框架PCANet。首先利用块匹配算法提取Ref与LR最相似的块PR和PL，PR和PL进行预处理后输入到网络中进行训练。PHi是网络第i层的输出。在数十万次迭代后网络趋近于最优解。
本文在三个基准数据集上进行了定性和定量的评估。结果显示：本文的方法在三个数据集上均取得最好的表现，PSNR平均结果要高于第二名至少0.3db以上。另外误差热力图显示本文的方法恢复的图像结果最接近于真值(ground truth)。这说明参考图的引入可以有效地辅助低分辨率图像重建。本文提出新的注意力通道机制来强化对不同信息的使用，以及采取渐进式网络结构充分挖掘参考图的高频信息。在今后的工作中，本文的算法可以将这一方法应用到更多的领域，如：基于参考图的去噪，基于参考图的去模糊等。

关键词: 基于参考图的超分辨率, 注意力机制, PCANet

Abstract: In recent years, the convolutional neural networks (CNNs) for single image super-resolution (SISR) are becoming more and more complex, and it is more challenging to improve the SISR performance. In contrast, the reference image guided super-resolution (RefSR) is an effective strategy to boost the SR (super-resolution) performance. In RefSR, the introduced high-resolution (HR) references can facilitate the high-frequency residual prediction process. According to the best of our knowledge, the existing CNN-based RefSR methods treat the features from the references and the low-resolution (LR) input equally by simply concatenating them together. However, the HR references and the LR inputs contribute differently to the final SR results. Therefore, we propose a progressive channel attention network (PCANet) for RefSR. There are two technical contributions in this paper. First, we propose a novel channel attention module (CAM), which estimates the channel weighting parameter by weightedly averaging the spatial features instead of using global averaging. Second, considering that the residual prediction process can be improved when the LR input is enriched with more details, we perform super-resolution progressively, which can take advantage of the reference images in multi-scales. Extensive quantitative and qualitative evaluations on three benchmark datasets, which represent three typical scenarios for RefSR, demonstrate that our method is superior to the state-of-the-art SISR and RefSR methods in terms of PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

Key words: reference-based super resolution, channel attention, progressive channel attention network (PCANet)

[1] Dong C, Loy C C, He K et al. Learning a deep convolutional network for image super-resolution. In Proc. the 13th European Conference on Computer Vision, September 2014, pp.184-199.
[2] Wang Z, Liu D, Yang J et al. Deep networks for image super-resolution with sparse prior. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.370-378.
[3] Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1646-1654.
[4] Lai W S, Huang J B, Ahuja N et al. Deep Laplacian pyramid networks for fast and accurate super-resolution. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.5835-5843.
[5] Tong T, Li G, Liu X et al. Image super-resolution using dense skip Conference on Computer Vision, October 2017, pp.4809-4817.
[6] Lim B, Son S, Kim H et al. Enhanced deep residual networks for single image super-resolution. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, July 2017, pp.1132-1140.
[7] Hu Y, Li J, Huang Y et al. Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Transactions on Circuits and Systems for Video Technology. doi:10.1109/TCSVT.2019.2915238.
[8] Zhang Y, Li K, Li K et al. Image super-resolution using very deep residual channel attention networks. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.294-310.
[9] Liu S, Gang R, Li C et al. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 5(4):391-401.
[10] Yue H, Sun X, Yang J et al. Landmark image superresolution by retrieving web images. IEEE Transactions on Image Processing, 2013, 22(12):4865-4878.
[11] Zhang Z, Wang Z, Lin Z et al. Image super-resolution by neural texture transfer. In Proc. the 2019 IEEE Conference on Computer Vision and Pattern Recognition, June 2019, pp.7982-7991.
[12] Zheng H, Ji M, Wang H et al. CrossNet:An end-to-end reference-based super resolution network using cross-scale warping. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.87-104.
[13] Li Y, Dong W, Shi G et al. Learning parametric distributions for image super-resolution:Where patch matching meets sparse coding. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.450-458.
[14] Liu J, Yang W, Zhang X et al. Retrieval compensated group structured sparsity for image super-resolution. IEEE Transactions on Multimedia, 2017, 19(2):302-316.
[15] Yang W, Xia S, Liu J et al. Reference-guided deep superresolution via manifold localized external compensation. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(5):1270-1283.
[16] Yue H, Liu J, Yang J et al. IENet:Internal and external patch matching ConvNet for web image guided denoising. IEEE Transactions on Circuits and Systems for Video Technology. doi:10.1109/TCSVT.2019.2930305.
[17] Zhang J, Gai D, Zhang X et al. Multi-example featureconstrained back-projection method for image superresolution. Computational Visual Media, 2017, 3(1):73-82.
[18] Zhao X, Wu Y, Tian J et al. Single image super-resolution via blind blurring estimation and anchored space mapping. Computational Visual Media, 2016, 2(1):71-85.
[19] Glasner D, Bagon S, Irani M. Super-resolution from a single image. In Proc. the 12th IEEE International Conference on Computer Vision, September 2009, pp.349-356.
[20] Freeman W T, Jones T R, Pasztor E C. Example-based super-resolution. IEEE Computer Graphics and Applications, 2002, 22(2):56-65.
[21] Yang J, Wright J, Huang T et al. Image super-resolution as sparse representation of raw image patches. In Proc. the 2008 IEEE Conference on Computer Vision and Pattern Recognition, June 2008.
[22] Yang J, Wright J, Huang T S et al. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11):2861-2873.
[23] Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1637-1645.
[24] Shi W, Caballero J, Huszár F et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1874-1883.
[25] Ledig C, Theis L, Huszár F et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.105-114.
[26] He K, Zhang X, Ren S et al. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.
[27] Goodfellow I, Pouget-Abadie J, Mirza M et al. Generative adversarial nets. In Proc. the 2014 Annual Conference on Neural Information Processing Systems, December 2014, pp.2672-2680.
[28] Johnson J, Alahi A, Li F F. Perceptual losses for realtime style transfer and super-resolution. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.694-711.
[29] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In Proc. the 2018 IEEE Conference on Computer Vision and Pattern Recognition, June 2018, pp.7132-7141.
[30] Mansimov E, Parisotto E, Ba J L et al. Generating images from captions with attention. arXiv:1511.02793, 2015. https://arxiv.org/abs/1511.02793, Jan. 2020.
[31] Kim J H, Choi J H, Cheon M et al. Ram:Residual attention module for single image super-resolution. arXiv:1811.12043, 2018. https://arxiv.org/pdf/1811.12043.pdf, Jan. 2020.
[32] Woo S, Park J, Lee J Y et al. CBAM:Convolutional block attention module. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.3-19.
[33] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2):91-110.
[34] Fischler M A, Bolles R C. Random sample consensus:A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981, 24(6):381-395.
[35] Kingma D P, Ba J. ADAM:A method for stochastic optimization. arXiv:1412.6980, 2014. https://arxiv.org/pdf/1412.6980.pdf, Jan. 2020.
[36] Cao Q, Shen L, Xie W et al. VGGFace2:A dataset for recognising faces across pose and age. In Proc. the 13th IEEE International Conference on Automatic Face&Gesture Recognition, May 2018, pp.67-74.
[37] Wang Z, Bovik A C, Sheikh H R et al. Image quality assessment:From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4):600-612.
[1] Jia-Ke Ge, Yan-Feng Chai, Yun-Peng Chai. WATuning：一种基于注意力机制的深度强化学习的工作负载感知调优系统[J]. 计算机科学技术学报, 2021, 36(4): 741-761.
[2] Chen-Chen Sun, De-Rong Shen. 面向深度实体匹配的混合层次网络[J]. 计算机科学技术学报, 2021, 36(4): 822-838.
[3] Sheng-Luan Hou, Xi-Kun Huang, Chao-Qun Fei, Shu-Han Zhang, Yang-Yang Li, Qi-Lin Sun, Chuan-Qing Wang. 基于深度学习的文本摘要研究综述[J]. 计算机科学技术学报, 2021, 36(3): 633-663.
[4] Yang Liu, Ruili He, Xiaoqian Lv, Wei Wang, Xin Sun, Shengping Zhang. 婴儿的年龄和性别容易被识别吗？[J]. 计算机科学技术学报, 2021, 36(3): 508-519.
[5] Yi-Ting Wang, Jie Shen, Zhi-Xu Li, Qiang Yang, An Liu, Peng-Peng Zhao, Jia-Jie Xu, Lei Zhao, Xun-Jie Yang. 基于搜索引擎丰富上下文信息的实体链接方法[J]. 计算机科学技术学报, 2020, 35(4): 724-738.
[6] Ying Li, Jia-Jie Xu, Peng-Peng Zhao, Jun-Hua Fang, Wei Chen, Lei Zhao. ATLRec:用于跨领域推荐的注意力对抗迁移学习网络[J]. 计算机科学技术学报, 2020, 35(4): 794-808.
[7] Chun-Yang Ruan, Ye Wang, Jiangang Ma, Yanchun Zhang, Xin-Tian Chen. 基于元路径注意力机制的异构网络对抗式嵌入[J]. 计算机科学技术学报, 2019, 34(6): 1217-1229.
[8] Qian Wang, You-Dong Ding. 基于局部加强PCAnet神经网络的细粒度车型识别方法[J]. , 2018, 33(2): 335-350.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] 李未;. A Structural Operational Semantics for an Edison Like Language(2)[J]. , 1986, 1(2): 42 -53 .
[2] 李万学;. Almost Optimal Dynamic 2-3 Trees[J]. , 1986, 1(2): 60 -71 .
[3] 冯玉琳;. Recursive Implementation of VLSI Circuits[J]. , 1986, 1(2): 72 -82 .
[4] 郑国梁; 李辉;. The Design and Implementation of the Syntax-Directed Editor Generator(SEG)[J]. , 1986, 1(4): 39 -48 .
[5] 龚振和;. On Conceptual Model Specification and Verification[J]. , 1987, 2(1): 35 -50 .
[6] 闵应骅;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[7] 乔香珍;. An Efficient Parallel Algorithm for FFT[J]. , 1987, 2(3): 174 -190 .
[8] 孙永强; 陆汝占; 黄小戎;. Termination Preserving Problem in the Transformation of Applicative Programs[J]. , 1987, 2(3): 191 -201 .
[9] 孟力明; 徐晓飞; 常会友; 陈光熙; 胡铭曾; 李生;. A Tree-Structured Database Machine for Large Relational Database Systems[J]. , 1987, 2(4): 265 -275 .
[10] 冯玉琳;. Hierarchical Protocol Analysis by Temporal Logic[J]. , 1988, 3(1): 56 -69 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn
总访问量：