›› 2017,Vol. 32 ›› Issue (3): 494-506.doi: 10.1007/s11390-017-1739-6

所属专题： Computer Graphics and Multimedia

• Special Section on Selected Paper from NPC 2011 • 上一篇    下一篇

提升空间一致性与边缘定位的图像自动上色方法

Wei Zhang1, Member, IEEE, Chao-Wei Fang1, Guan-Bin Li2,*, Member, IEEE   

  1. 1. Department of Computer Science, The University of Hong Kong, Hong Kong, China;
    2. School of Data and Computer Science, Sun Yat-sen University, Guangzhou 510006, China
  • 收稿日期:2016-12-25 修回日期:2017-02-26 出版日期:2017-05-05 发布日期:2017-05-05
  • 通讯作者: Guan-Bin Li E-mail:liguanbin@mail.sysu.edu.cn
  • 作者简介:Wei Zhang is a Ph.D. candidate at the Department of Computer Science, The University of Hong Kong, Hong Kong. He received his B.E. degree in automation from Chongqing University, Chongqing, in 2010, and M.S. degree in pattern recognition and artificial intelligence from Huazhong University of Science and Technology, Wuhan, in 2013. His research interest covers image colorization and boundary detection.
  • 基金资助:

    This work was partially supported by Hong Kong Research Grants Council under General Research Funds (HKU17209714).

Automatic Colorization with Improved Spatial Coherence and Boundary Localization

Wei Zhang1, Member, IEEE, Chao-Wei Fang1, Guan-Bin Li2,*, Member, IEEE   

  1. 1. Department of Computer Science, The University of Hong Kong, Hong Kong, China;
    2. School of Data and Computer Science, Sun Yat-sen University, Guangzhou 510006, China
  • Received:2016-12-25 Revised:2017-02-26 Online:2017-05-05 Published:2017-05-05
  • Contact: Guan-Bin Li E-mail:liguanbin@mail.sysu.edu.cn
  • About author:Wei Zhang is a Ph.D. candidate at the Department of Computer Science, The University of Hong Kong, Hong Kong. He received his B.E. degree in automation from Chongqing University, Chongqing, in 2010, and M.S. degree in pattern recognition and artificial intelligence from Huazhong University of Science and Technology, Wuhan, in 2013. His research interest covers image colorization and boundary detection.
  • Supported by:

    This work was partially supported by Hong Kong Research Grants Council under General Research Funds (HKU17209714).

PDF (PC)

1062

给灰度图像上色是计算机图形学中的一个重要的问题并且存在很多的应用。最近的研究将图像全自动上色看作一个对图像中每个像素颜色值进行预测的问题，同时利用了深度卷积神经网络，使得图像全自动上色技术取得了显著的进步。然而，当前的结果距离高质量的全自动上色依旧存在很大的距离。具体来说，在目前的算法结果中，普遍存在两类问题，一是色彩平滑的区域内部常常无法得到较为一致的上色结果，二是在图像内部不同区域的边界处，会出现颜色溢出。为了解决这些问题，我们提出了新的全自动图像上色的算法流程，其中包含了一个边界指导的条件随机场以及基于卷积神经网络的颜色变换作为后处理步骤。此外，因为每张图片往往存在多种合理的上色方案，对算法上色结果的优劣进行自动评价也是一项有挑战性的工作。我们进一步提出两个全新的自动评价体系，使得可以从区域内上色的一致性和边界保持这两个方面对上色结果进行有效的评价。实验结果表明，在多种评价体系下，我们提出的算法能够得到更好的自动上色结果。

Abstract: Grayscale image colorization is an important computer graphics problem with a variety of applications. Recent fully automatic colorization methods have made impressive progress by formulating image colorization as a pixel-wise prediction task and utilizing deep convolutional neural networks. Though tremendous improvements have been made, the result of automatic colorization is still far from perfect. Specifically, there still exist common pitfalls in maintaining color consistency in homogeneous regions as well as precisely distinguishing colors near region boundaries. To tackle these problems, we propose a novel fully automatic colorization pipeline which involves a boundary-guided CRF (conditional random field) and a CNN-based color transform as post-processing steps. In addition, as there usually exist multiple plausible colorization proposals for a single image, automatic evaluation for different colorization methods remains a challenging task. We further introduce two novel automatic evaluation schemes to efficiently assess colorization quality in terms of spatial coherence and localization. Comprehensive experiments demonstrate great quality improvement in results of our proposed colorization method under multiple evaluation metrics.

[1] Levin A, Lischinski D, Weiss Y. Colorization using optimization. ACM Transactions on Graphics (TOG), 2004, 23(3): 689-694.

[2] Huang Y C, Tung Y S, Chen J C, Wang S W, Wu J L. An adaptive edge detection based colorization algorithm and its applications. In Proc. the 13th Annual ACM International Conference on Multimedia, Nov. 2005, pp.351-354.

[3] Luan Q, Wen F, Cohen-Or D, Liang L, Xu Y Q, Shum H Y. Natural image colorization. In Proc. the 18th Eurographics Conference on Rendering Techniques, Jun. 2007, pp.309-320.

[4] Qu Y,Wong T T, Heng P A. Manga colorization. ACM Transactions on Graphics (TOG), 2006, 25(3): 1214-1220.

[5] Zhao H L, Nie G Z, Li X J, Jin X G, Pan Z G. Structureaware nonlocal optimization framework for image colorization. Journal of Computer Science and Technology, 2015, 30(3): 478-488.

[6] Sheng B, Sun H, Magnor M, Li P. Video colorization using parallel optimization in feature space. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(3): 407-417.

[7] Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images. ACM Transactions on Graphics (TOG), 2002, 21(3): 277-280.

[8] Irony R, Cohen-Or D, Lischinski D. Colorization by example. In Proc. Eurographics Symp. Rendering Techqiques, June 29-July 1, 2005, pp.201-210.

[9] Charpiat G, Hofmann M, Schölkopf B. Automatic image colorization via multimodal predictions. In Proc. the 10th European Conference on Computer Vision, Oct. 2008, pp.126-139.

[10] Liu X, Wan L, Qu Y, Wong T T, Lin S, Leung C S, Heng P A. Intrinsic colorization. ACM Transactions on Graphics (TOG), 2008, 27(5): 152:1-152:9.

[11] Gupta R K, Chia A Y S, Rajan D, Ng E S et al. Image colorization using similar images. In Proc. the 20th ACM International Conference on Multimedia, Oct.29-Nov.2, 2012, pp.369-378.

[12] Jin S Y, Choi H J, Tai Y W. A randomized algorithm for natural object colorization. Computer Graphics Forum, 2014, 33(2): 205-214.

[13] Chia A Y S, Zhuo S, Gupta R K, Tai Y W, Cho S Y, Tan P, Lin S. Semantic colorization with Internet images. ACM Transactions on Graphics (TOG), 2011, 30(6): 156:1-156:8.

[14] Deshpande A, Rock J, Forsyth D. Learning large-scale automatic image colorization. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.567-575.

[15] Li X, Zhao H, Nie G, Huang H. Image recoloring using geodesic distance based color harmonization. Computational Visual Media, 2015, 1(2): 143-155.

[16] Cheng Z, Yang Q, Sheng B. Deep colorization. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.415-423.

[17] Dahl R. Automatic colorization. http://tinyclouds.org/colorize/, Aug. 2016.

[18] Larsson G, Maire M, Shakhnarovich G. Learning representations for automatic colorization. In Proc. European Conference on Computer Vision, Oct. 2016, pp.577-593.

[19] Zhang R, Isola P, Efros A A. Colorful image colorization. In Proc. European Conference on Computer Vision, Oct. 2016, pp.649-666.

[20] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. https://arxiv.org/abs/1409.1556, Aug. 2016.

[21] Hariharan B, Arbeláez P, Girshick R, Malik J. Hypercolumns for object segmentation and fine-grained localization. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.447-456.

[22] Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.1520-1528.

[23] Li G, Yu Y. Deep contrast learning for salient object detection. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.478-487.

[24] Li G, Yu Y. Visual saliency based on multiscale deep features. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.5455-5463.

[25] Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proc. Advances in Neural Information Processing Systems, Dec. 2015, pp.91-99.

[26] Xie S, Tu Z. Holistically-nested edge detection. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.1395-1403.

[27] Iizuka S, Simo-Serra E, Ishikawa H. Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (TOG), 2016, 35(4): 110:1-110:11.

[28] Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.1520-1528.

[29] Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122, 2015. https://arxiv.org/abs/1511.07122, Aug. 2016.

[30] Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(11): 1222- 1239.

[31] He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.1026-1034.

[32] Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T. Caffe: Convolutional architecture for fast feature embedding. In Proc. the 22nd ACM International Conference on Multimedia, Nov. 2014, pp.675-678.

[33] Felzenszwalb P F, Huttenlocher D P. Efficient graph-based image segmentation. International Journal of Computer Vision, 2004, 59(2): 167-181.

[34] Arbeláez P, Pont-Tuset J, Barron J T, Marques F, Malik J. Multiscale combinatorial grouping. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp.328-335.

[35] Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3): 211-252.
No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] 刘明业; 洪恩宇;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[2] 陈世华;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[3] 高庆狮; 张祥; 杨树范; 陈树清;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[4] 金兰; 杨元元;. A Modified Version of Chordal Ring[J]. , 1986, 1(3): 15 -32 .
[5] 陈肇雄; 高庆狮;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[6] 黄河燕;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[7] 闵应骅; 韩智德;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[8] 唐同诰; 招兆铿;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[9] 闵应骅;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[10] 朱鸿;. Some Mathematical Properties of the Functional Programming Language FP[J]. , 1987, 2(3): 202 -216 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn
总访问量：