Journal of Computer Science and Technology ›› 2019, Vol. 34 ›› Issue (6): 1294-1306.doi: 10.1007/s11390-019-1976-y

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Computer Graphics and Multimedia • Previous Articles     Next Articles

Artistic Augmentation of Photographs with Droplets

Mo-Han Zhang1, Jin-Hui Yu1,2, Kang Zhang3,4, Jun-Song Zhang5   

  1. 1 State Key Laboratory of CAD & CG, Zhejiang University, Hangzhou 310027, China;
    2 Department of Computer Science, Harbin Finance University, Harbin 150030, China;
    3 Department of Computer Science, The University of Texas at Dallas, Richardson, TX 75080-3021, U.S.A.;
    4 Faculty of Information Technology, Macau University of Science and Technology, Macau 999078, China;
    5 National Engineering Research Center for E-Learning, Central China Normal University, Wuhan 430079, China
  • Received:2019-04-09 Revised:2019-08-29 Online:2019-11-16 Published:2019-11-16
  • About author:Mo-Han Zhang received his B.Sc. degree in software engineering from Sichuan University, Chengdu, in 2013. Currently, he is working toward his Ph.D. degree at the State Key Laboratory of CAD & CG, Zhejiang University, Hangzhou. His research interests include augmented reality, artistic augmentation, artificial intelligence and nonphotorealistic rendering.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China under Grant Nos. 61772463 and 61772440, the Aeronautical Science Foundation of China under Grant No. 20165168007, and the National Key Research and Development Program of China under Grant No. 2018YFB1700900.

Artistic augmentation of photographs with water droplets aims at generating aesthetic yet realistic images, and thus differs from traditional augmented reality in two aspects. One difference lies in the adoption of a new image as the environment map in order to render reflected or refracted effects on the surface of inserted water droplets. The other difference is in modeling of water droplets including hanging droplets and resting droplets. These differences raise two research challenges:1) how to adjust the brightness and colors of the new environment map to maintain visual consistency between the new environment map and the original input image; 2) how to model hanging and resting droplets aesthetically. This paper proposes a framework that addresses these two challenges and demonstrates the effectiveness of our framework by generating example augmented images.

Key words: artistic augmentation; augmented reality; illumination estimation; droplet modeling; image-based modeling;

[1] Kronander J, Banterle F, Gardner A, Miandji E, Unger J. Photorealistic rendering of mixed reality scenes. Comput. Graph. Forum, 2015, 34(2):643-665.
[2] Hartley R I, Zisserman A. Multiple View Geometry in Computer Vision (2nd edition). Cambridge University Press, 2004.
[3] Schmalstieg D, Höllerer T. Augmented Reality:Theory and Practice (1st edition). Addison-Wesley Professional, 2016.
[4] van Krevelen D W F, Poelman R. A survey of augmented reality technologies, applications and limitations. The International Journal of Virtual Reality, 2010, 9(2):1-20.
[5] Rabbi I, Ullah S. A survey of augmented reality challenges and tracking. ACTA GRAPHICA, 2013, 24(1/2):29-46.
[6] Debevec P. Rendering synthetic objects into real scenes:Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. the 25th Annual Conference on Computer Graphics and Interactive Techniques, July 1998, pp.189-198.
[7] Stumpfel J, Tchou C, Jones A, Hawkins T, Wenger A, Debevec P. Direct HDR capture of the sun and sky. In Proc. the 3rd International Conference on Computer Graphics, November 2004, pp.145-149.
[8] Banterle F, Ledda P, Debattista K, Chalmers A. Inverse tone mapping. In Proc. the 4th International Conference on Computer Graphics and Interactive Techniques, November 2006, pp.349-356.
[9] Banterle F, Debattista K, Artusi A, Pattanaik S, Myszkowski K, Ledda P, Chalmers A. High dynamic range imaging and low dynamic range expansion for generating HDR content. Computer Graphics Forum, 2009, 28(8):2343-2367.
[10] Pellacini F. Envylight:An interface for editing natural illumination. ACM Transactions on Graphics, 2010, 29(4):Article No. 34.
[11] Debevec P, Graham P, Busch J, Bolas M. A single-shot light probe. In Proc. the 39th International Conference on Computer Graphics and Interactive Techniques, August 2012, Article No. 10.
[12] Reinhard E, Ashikhmin M, Gooch B, Shirley P. Color transfer between images. IEEE Computer Graphics and Applications, 2001, 21(5):34-41.
[13] Reinhard E, Akyuz O A, Colbert M, Hughes C, O'Connor M. Real-time color blending of rendered and captured video. In Proc. Interservice/Industry Training, Simulation and Education Conference, Dec. 2004, Article No. 1502.
[14] Khan E A, Reinhard E, Fleming R W, Bülthoff H H. Imagebased material editing. ACM Transactions on Graphics, 2006, 25(3):654-663.
[15] Lopez-Moreno J, Hadap S, Reinhard E, Gutierrez D. Compositing images through light source detection. Computers & Graphics, 2010, 34(6):698-707.
[16] Karsch K, Hedau V, Forsyth D, Hoiem D. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics, 2011, 30(6):Article No. 157.
[17] Karsch K, Sunkavalli K, Hadap S, Carr N, Jin H, Fonte R, Sittig M, Forsyth D. Automatic scene inference for 3D object compositing. ACM Transactions on Graphics, 2014, 33(3):Article No. 32.
[18] Grosse R, Johnson M K, Adelson E H, Freeman W T. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In Proc. the 12th IEEE International Conference on Computer, September 2009, pp.2335-2342.
[19] Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2012, 34(11):2274-2282.
[20] Lalonde J F, Efros A A, Narasimhan S G. Webcam clip art:Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics, 2009, 28(5):Article No. 131.
[21] Perez R, Seals R, Michalsky J. All-weather model for sky luminance distribution-Preliminary configuration and validation. Solar Energy, 1993, 50(3):235-245.
[22] Lalonde J F, Efros A A, Narasimhan S G. Estimating natural illumination from a single outdoor image. In Proc. the 12th IEEE International Conference on Computer Vision, September 2009, pp.183-190.
[23] Lalonde J F, Efros A A, Narasimhan S G. Estimating the natural illumination conditions from a single outdoor image. International Journal of Computer Vision, 2012, 98(2):123-145.
[24] Lalonde J F. Understanding and recreating visual appearance under natural illumination[Ph.D. Thesis]. Carnegie Mellon University, 2011.
[25] Liu Y, Qin X, Xu S, Nakamae E, Peng Q. Light source estimation of outdoor scenes for mixed reality. The Visual Computer, 2009, 25(5/6/7):637-646.
[26] Liu Y, Granier X. Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(4):573-580.
[27] Xing G, Zhou X, Peng Q, Liu Y, Qin X. Lighting simulation of augmented outdoor scene based on a legacy photograph. Computer Graphics Forum, 2013, 32(7):101-110.
[28] Blinn J F. A generalization of algebraic surface drawing. ACM Trans. Graph., 1982, 1(3):235-256.
[29] O'Brien J F, Hodgins J K. Dynamic simulation of splashing fluids. In Proc. the 1995 Computer Animation, April 1995, pp.198-205.
[30] Murta A, Miller J. Modelling and rendering liquids in motion. In Proc. the 7th International Conference in Central Europe on Computer Graphics, Visualization and Interactive Digital Media, February 1999, pp.194-201.
[31] Fournier P, Habibi A, Poulin P. Simulating the flow of liquid droplets. In Proc. the 1998 Graphics Interface Conference, June 1998, pp.133-142.
[32] Kaneda K, Kagawa T, Yamashita H. Animation of water droplets on a glass plate. In Proc. the 5th International Workshop on Computer Animation, June 1993, pp.177-189.
[33] Kaneda K, Ikeda S, Yamashita H. Animation of water droplets moving down a surface. The Journal of Visualization and Computer Animation, 1999, 10(1):15-26.
[34] Yu Y J, Jung H Y, Cho H G. A new water droplet model using metaball in the gravitational field. Computers & Graphics, 1999, 23(2):213-222.
[35] Tong R, Kaneda K, Yamashita H. A volume-preserving approach for modeling and animating water flows generated by metaballs. The Visual Computer, 2002, 18(8):469-480.
[36] Wolff L B. Using polarization to separate reflection components. In Proc. the 1989 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 1989, pp.363-369.
[37] Lalonde J F, Narasimhan S G, Efros A A. What do the sun and the sky tell us about the camera? International Journal of Computer Vision, 2010, 88(1):24-51.
[38] Hold-Geoffroy Y, Sunkavalli K, Hadap S, Gambaretto E, Lalonde J F. Deep outdoor illumination estimation. In Proc. the 2017 IEEE International Conference on Computer Vision and Pattern Recognition, July 2017, pp.2373-2382.
[39] Gardner M, Sunkavalli K, Yumer E, Shen X, Gambaretto E, Gagné C, Lalonde J F. Learning to predict indoor illumination from a single image. arXiv:1704.00090, 2017., June 2019.
[40] Georgoulis S, Rematas K, Ritschel T, Fritz M, Tuytelaars T, Gool L V. What is around the camera? In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017, pp.5180-5188.
[41] Hold-Geoffroy Y, Sunkavalli K, Eisenmann J, Fisher M, Gambaretto E, Hadap S, Lalonde J. A perceptual measure for deep single image camera calibration. arXiv:1712.01259, 2017., June 2019.
[42] Xing G, Liu Y, Peng Q, Qin X. On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In Proc. the 12th International Conference on Computer-Aided Design and Computer Graphics, September 2011, pp.439-442.
[43] Xing G, Liu Y, Qin X, Peng Q. A practical approach for real-time illumination estimation of outdoor videos. Computers & Graphics, 2012, 36(7):857-865.
[44] Boivin S, Gagalowicz A. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. the 28th Annual Conference on Computer Graphics and Interactive Techniques, August 2001, pp.107-116.
[45] Bousseau A, Paris S, Durand F. User-assisted intrinsic images. ACM Trans. Graph., 2009, 28(5):Article No. 130.
[46] MacRobert T. Spherical Harmonics:An Elementary Treatise on Harmonic Functions with Applications (3rd rev. edition). Pergamon Press, 1967.
[47] Sillion F X, Arvo J R, Westin S H, Greenberg D P. A global illumination solution for general reflectance distributions. ACM SIGGRAPH Comput. Graph., 1991, 25(4):187-196.
[48] Ramamoorthi R, Hanrahan P. On the relationship between radiance and irradiance:Determining the illumination from images of a convex Lambertian object. Journal of the Optical Society of America A, Optics, Image Science, and Vision, 2001, 18(10):2448-2459.
[49] Ramamoorthi R, Hanrahan P. An efficient representation for irradiance environment maps. In Proc. the 28th Annual Conference on Computer Graphics and Interactive Techniques, August 2001, pp.497-500.
[50] Ren H, Xu S, Wu S T. Effects of gravity on the shape of liquid droplets. Optics Communications, 2010, 283(17):3255-3258.
[51] Dixit S, Pincus A, Guo B, Faris G. Droplet shape analysis and permeability studies in droplet lipid bilayers. Langmuir:The ACS Journal of Surfaces and Colloids, 2012, 28(19):7442-7451.
[52] Sahin S, Bliznyuk O, Cordova R A, Schröen K. Microfluidic EDGE emulsification:The importance of interface interactions on droplet formation and pressure stability. Scientific Reports, 2016, 6:Article No. 26407.
[53] Li Z, Wang S, Yu J, Ma K L. Restoration of brick and stone relief from single rubbing images. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(2):177-187.
[54] Zhang T, Zhang L, Yu J. Computer generation of 3D inscriptions from 2D images of Chinese calligraphy. Chinese Journal of Computers, 2014, 37(11):2380-2388. (in Chinese)
[55] Hamlyn D W. The Psychology of Perception:A Philosophical Examination of Gestalt Theory and Derivative Theories of Perception (1st edition). Routledge, 1957.
[1] Hao-Ren Wang, Juan Lei, Ao Li, Yi-Hong Wu. A Geometry-Based Point Cloud Reduction Method for Mobile Augmented Reality System [J]. Journal of Computer Science and Technology, 2018, 33(6): 1164-1177.
[2] Bin Liu, Kun Xu, Ralph R. Martin. Static Scene Illumination Estimation from Video with Applications [J]. , 2017, 32(3): 430-442.
[3] Liu-Xin Zhang (张柳新), Member, CCF, Ming-Tao Pei (裴明涛), Member, CCF and Yun-De Jia (贾云得), Senior Member, CCF. Multiview Visibility Estimation for Image-Based Modeling [J]. , 2011, 26(6): 1000-1010.
[4] Yu Li, Yong-Tian Wang, and Yue Liu. Fiducial Marker Based on Projective Invariant for Augmented Reality [J]. , 2007, 22(6): 890-897 .
Full text



No Suggested Reading articles found!

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved