1. Department of Film and Television Engineering, Shanghai University, Shanghai 200072, China;
2. Shanghai Engineering Research Center of Motion Picture Special Effects, Shanghai 200072, China;
3. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Abstract A number of edge-aware filters can efficiently boost the appearance of an image by detail decomposition and enhancement. However, they often fail to produce photographic enhanced appearance due to some visible artifacts, especially noise, halos and unnatural contrast. The essential reason is that the guidance and constraint of high-quality appearance aren't enough sufficient in the process of enhancement. Thus our idea is to train a detail dictionary from a lot of high-quality patches in order to constrain and control the entire appearance enhancement. In this paper, we propose a novel learning-based enhancement method for photographic appearance, which includes two main stages: dictionary training and sparse reconstruction. In the training stage, we construct a training set of detail patches by extracting from some high-quality photos, and then train an overcomplete detail dictionary by iteratively minimizing an L1-norm energy function. In the reconstruction stage, we employ the trained dictionary to reconstruct the boosted detail layer, and further formalize a gradient-guided optimization function to improve the local coherence between patches. Moreover, we propose two evaluation metrics to measure the performance of appearance enhancement. The final experimental results have demonstrated the effectiveness of our learning-based enhancement method.
This work was supported by the National Natural Science Foundation of China under Grant Nos.61303093,61402278,and 61472245,the Innovation Program of the Science and Technology Commission of Shanghai Municipality of China under Grant No.16511101300,the Natural Science Foundation of Shanghai under Grant No.14ZR1415800,and the Gaofeng Film Discipline Grant of Shanghai Municipal Education Commission.
Corresponding Authors: 10.1007/s11390-017-1733-z
About author: Zhi-Feng Xie received his Ph.D. degree in computer application technology from Shanghai Jiao Tong University, Shanghai, in 2013. He was a research assistant at the Department of Computer Science, City University of Hong Kong, Hong Kong, in 2011. He is now an assistant professor with Shanghai University, Shanghai. His research interests include image/video editing, computer graphics, and digital media technology.
Cite this article:
Zhi-Feng Xie, Shi Tang, Dong-Jin Huang, You-Dong Ding, Li-Zhuang Ma.Photographic Appearance Enhancement via Detail-based Dictionary Learning[J] Journal of Computer Science and Technology, 2017,V32(3): 417-429
 Paris S, Durand F. A fast approximation of the bilateral filter using a signal processing approach. In Proc. the 9th ECCV, May 2006, pp.568-580. He K M, Sun J, Tang X O. Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1397-1409. Farbman Z, Fattal R, Lischinski D, Szeliski R. Edgepreserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics, 2008, 27(3): 67:1-67:10. Paris S, Hasinoff S, Kautz J. Local Laplacian filters: Edgeaware image processing with a Laplacian pyramid. ACM Transactions on Graphics, 2011, 30(4): 68:1-68:12. Xu L, Lu C W, Xu Y, Jia J Y. Image smoothing via L0 gradient minimization. ACM Transactions on Graphics, 2011, 30(6): 174:1-174:12. Zhang Q, Shen X Y, Xu L, Jia J Y. Rolling guidance filter. In Proc. the 13th ECCV, Sept. 2014, pp.815-830. Wright J, Ma Y, Mairal J, Sapiro G, Huang T S, Yan S. Sparse representation for computer vision and pattern recognition. Proceedings of the IEEE, 2010, 98(6): 1031- 1044. Aharon M, Elad M, Bruckstein A. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 2006, 54(11): 4311-4322. Perona P, Malik J. Scale-space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1990, 12: 629-639. Tomasi C, Manduchi R. Bilateral filtering for gray and color images. In Proc. ICCV, Jan. 1998, pp.839-846. Durand F, Dorsey J. Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics, 2002, 21(3): 257-266. Bae S, Paris S, Durand F. Two-scale tone management for photographic look. ACM Transactions on Graphics, 2006, 25(3): 637-645. Weiss B. Fast median and bilateral filtering. ACM Transactions on Graphics, 2006, 25(3): 519-526. Chen J W, Paris S, Durand F. Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics, 2007, 26(3): 103:1-103:9. Porikli F. Constant time O(1) bilateral filtering. In Proc. CVPR, June 2008. Yang Q X, Tan K H, Ahuja N. Real-time O(1) bilateral filtering. In Proc. CVPR, June 2009, pp.557-564. Fattal R. Edge-avoiding wavelets and their applications. ACM Transactions on Graphics, 2009, 28(3): 22:1-22:10. Subr K, Soler C, Durand F. Edge-preserving multiscale image decomposition based on local extrema. ACM Transactions on Graphics, 2009, 28(5): 147:1-147:9. Gastal E, Oliveira M. Domain transform for edge-aware image and video processing. ACM Transactions on Graphics, 2011, 30(4): 69:1-69:12. Li X Y, Gu Y, Hu S M, Martin R. Mixed-domain edgeaware image manipulation. IEEE Transactions on Image Processing, 2013, 22(5): 1915-1925. Wang H, Cao J J, Liu X P, Wang J M, Fan T R, Hu J P. Least-squares images for edge-preserving smoothing. Computational Visual Media, 2015, 1(1): 27-35. Shao P, Ding S H, Ma L Z, Wu Y S, Wu Y J. Edgepreserving image decomposition via joint weighted least squares. Computational Visual Media, 2015, 1(1): 37-47. Xu L, Ren J, Yan Q, Liao R J, Jia J Y. Deep edge-aware filters. In Proc. the 32nd ICML, July 2015, pp.1669-1678. Gonzalez R, Woods R. Digital Image Processing (3rd edition). Upper Saddle River, NJ, USA: Prentice-Hall, 2006. Stark J A. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on Image Processing, 2000, 9(5): 889-896. Arici T, Dikbas S, Altunbasak Y. A histogram modification framework and its application for image contrast enhancement. IEEE Transactions on Image Processing, 2009, 18(9): 1921-1935. Celik T, Tjahjadi T. Contextual and variational contrast enhancement. IEEE Transactions on Image Processing, 2011, 20(12): 3431-3441. Celik T. Two-dimensional histogram equalization and contrast enhancement. Pattern Recognition, 2012, 45(10): 3810-3824. Fattal R, Agrawala M, Rusinkiewicz S. Multiscale shape and detail enhancement from multi-light image collections. ACM Transactions on Graphics, 2007, 26(3): 51:1-51:10. Joshi N, Matusik W, Adelson E, Kriegman D. Personal photo enhancement using example images. ACM Transactions on Graphics, 2010, 29(2): 12:1-12:15. Kang S B, Kapoor A, Lischinski D. Personalization of image enhancement. In Proc. the 23rd CVPR, June 2010, pp.1799- 1806. Caicedo J C, Kapoor A, Kang S B. Collaborative personalization of image enhancement. In Proc. the 24th CVPR, June 2011, pp.249-256. Bychkovsky V, Paris S, Chan E, Durand F. Learning photographic global tonal adjustment with a database of input/output image pairs. In Proc. the 24th CVPR, June 2011, pp.97-104. Kaufman L, Lischinski D, Werman M. Content-aware automatic photo enhancement. Computer Graphics Forum, 2012, 31(8): 2528-2540. Lischinski D, Farbman Z, Uyttendaele M, Szeliski R. Interactive local adjustment of tonal values. ACM Transactions on Graphics, 2006, 25(3): 646-653. Zhang F, Zhang X, Qin X Y, Zhang C M. Enlarging image by constrained least square approach with shape preserving. Journal of Computer Science and Technology, 2015, 30(3): 489-498. Liu Q, Chen M Y, Zhou D H. Single image haze removal via depth-based contrast stretching transform. Science China: Information Sciences, 2015, 58(1): 1-17. Zhang Q, Nie Y W, Zhang L, Xiao C X. Underexposed video enhancement via perception-driven progressive fusion. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(6): 1773-1785. Cheng M M, Hou Q B, Zhang S H, Rosin P. Intelligent visual media processing: When graphics meets vision. Journal of Computer Science and Technology, 2017, 32(1): 110-121. DeCarlo D, Santella A. Stylization and abstraction of photographs. ACM Transactions on Graphics, 2002, 21(3): 769-776. Winnemoller H, Olsen S, Gooch B. Real-time video abstraction. ACM Transactions on Graphics, 2006, 25(3): 1221- 1226. Echevarria J, Wilensky G, Krishnaswamy A, Kim B, Gutierrez D. Computational simulation of alternative photographic processes. Computer Graphics Forum, 2013, 32(4): 7-16. Son M J, Lee Y J, Kang H, Lee S Y. Art-photographic detail enhancement. Computer Graphics Forum, 2014, 33(2): 391-400. Aydin T O, Smolic A, Gross M. Automated aesthetic analysis of photographic images. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(1): 31-42. Mairal J, Sapiro G, Elad M. Learning multiscale sparse representations for image and video restoration. Multiscale Modeling & Simulation, 2008, 7(1): 214-241. Dong W, Zhang L, Shi G, Li X. Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing, 2013, 22(4): 1620-1630. Yang J, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. Zhang Y, Liu J, Yang W, Guo Z. Image super-resolution based on structure-modulated sparse representation. IEEE Transactions on Image Processing, 2015, 24(9): 2797-2810. Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 2006, 15(12): 3736-3745. Zeng X, Bian W, Liu W, Shen J, Tao D. Dictionary pair learning on Grassmann manifolds for image denoising. IEEE Transactions on Image Processing, 2015, 24(11): 4556-4569.