›› 2016, Vol. 31 ›› Issue (3): 463-478.doi: 10.1007/s11390-016-1640-8

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Special Section of CVM 2016 • Previous Articles     Next Articles

View-Aware Image Object Compositing and Synthesis from Multiple Sources

Xiang Chen1, Member, ACM, Wei-Wei Xu1, Member, IEEE, Sai-Kit Yeung2, Member, IEEE, and Kun Zhou1,*, Fellow, IEEE   

  1. 1 State Key Laboratory of Computer Aided Design and Computer Graphics, Zhejiang University Hangzhou 310058, China;
    2 Vision, Graphics and Computational Design Group, Singapore University of Technology and Design Singapore 487372, Singapore
  • Received:2015-11-28 Revised:2016-03-07 Online:2016-05-05 Published:2016-05-05
  • Contact: Kun Zhou E-mail:kunzhou@acm.org
  • Supported by:

    This work is partially supported by the National Natural Science Foundation of China under Grant Nos. 61272305, 61303136, 61272392, and 61322204, and the National Program for Special Support of Eminent Professionals of China.

PDF

1087

Cited

2

Image compositing is widely used to combine visual elements from separate source images into a single image. Although recent image compositing techniques are capable of achieving smooth blending of the visual elements from different sources, most of them implicitly assume the source images are taken in the same viewpoint. In this paper, we present an approach to compositing novel image objects from multiple source images which have different viewpoints. Our key idea is to construct 3D proxies for meaningful components of the source image objects, and use these 3D component proxies to warp and seamlessly merge components together in the same viewpoint. To realize this idea, we introduce a coordinateframe based single-view camera calibration algorithm to handle general types of image objects, a structure-aware cuboid optimization algorithm to get the cuboid proxies for image object components with correct structure relationship, and finally a 3D-proxy transformation guided image warping algorithm to stitch object components. We further describe a novel application based on this compositing approach to automatically synthesize a large number of image objects from a set of exemplars. Experimental results show that our compositing approach can be applied to a variety of image objects, such as chairs, cups, lamps, and robots, and the synthesis application can create novel image objects with significant shape and style variations from a small set of exemplars.

[1] Perez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22(3): 313-318.

[2] Jia J, Sun J, Tang C K, Shum H Y. Drag-and-drop pasting. ACM Transactions on Graphics, 2006, 25(3): 631-637.

[3] Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lischinski D. Coordinates for instant image cloning. ACM Transactions on Graphics, 2009, 28(3): Article No. 67.

[4] Tao MW, Johnson M K, Paris S. Error-tolerant image compositing. In Proc. the 11th European Conference on Computer Vision, Sept. 2010, pp.31-44.

[5] Sunkavalli K, Johnson M K, Matusik W, Pfister H. Multiscale image harmonization. ACM Transactions on Graphics, 2010, 29(4): Article No. 125.

[6] Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M. Interactive digital photomontage. ACM Transactions on Graphics, 2004, 23(3): 294-302.

[7] Rother C, Kumar S, Kolmogorov V, Blake A. Digital tapestry [automatic image synthesis]. In Proc. IEEE CVPR, June 2005, pp.589-596.

[8] Rother C, Bordeaux L, Hamadi Y, Blake A. AutoCollage. ACM Transactions on Graphics, 2006, 25(3): 847-852.

[9] Wang J, Quan L, Sun J, Tang X, Shum H Y. Picture collage. In Proc. IEEE CVPR, June 2006, pp.347-354.

[10] Chen T, Cheng M M, Tan P, Shamir A, Hu S M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics, 2009, 28(5): 124:1-124:10.

[11] Eitz M, Richter R, Hildebrand K, Boubekeur T, Alexa M. Photosketcher: Interactive sketch-based image synthesis. IEEE Computer Graphics and Applications, 2011, 31(6): 56-66.

[12] Kalogerakis E, Chaudhuri S, Koller D, Koltun V. A probabilistic model for component-based shape synthesis. ACM Trans. Graph., 2012, 31(4): 55:1-55:11.

[13] Xu K, Zhang H, Cohen-Or D, Chen B. Fit and diverse: Set evolution for inspiring 3D shape galleries. ACM Trans. Graph., 2012, 31(4): 57:1-57:10.

[14] Burt P J, Adelson E H. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 1983, 2(4): 217-236.

[15] Ogden J M, Adelson E H, Bergen J R, Burt P J. Pyramidbased computer graphics. RCA Engineer, 1985, 30(5): 4-15.

[16] Porter T, Duff T. Compositing digital images. ACM SIGGRAPH Comput. Graph., 1984, 18(3): 253-259.

[17] Xue S, Agarwala A, Dorsey J, Rushmeier H. Understanding and improving the realism of image composites. ACM Transactions on Graphics, 2012, 31(4): Article No. 84.

[18] Diakopoulos N, Essa I, Jain R. Content based image synthesis. In Proc. the 3rd CIVR, July 2004, pp.299-307.

[19] Johnson M, Brostow G J, Shotton J et al. Semantic photo synthesis. Computer Graphics Forum, 2006, 25(3): 407-413.

[20] Lalonde J F, Hoiem D, Efros A A, Rother C, Winn J, Criminisi A. Photo clip art. ACM Transactions on Graphics, 2007, 26(3): Article No. 3.

[21] Hall P, Cai H, Wu Q, Corradi T. Cross-depiction problem: Recognition and synthesis of photographs and artwork. Computational Visual Media, 2015, 1(2): 91-103.

[22] Huang H, Zhang L, Zhang H C. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): Article No. 155.

[23] Yu Z, Lu L, Guo Y, Fan R, Liu M, Wang W. Content-aware photo collage using circle packing. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(2): 182-195.

[24] Risser E, Han C, Dahyot R, Grinspun E. Synthesizing structured image hybrids. ACM Transactions on Graphics, 2010, 29(4): Article No. 85.

[25] Carroll R, Agarwala A, Agrawala M. Image warps for artistic perspective manipulation. ACM Transactions on Graphics, 2010, 29(4): Article No. 127.

[26] Zheng Y, Chen X, Cheng M M, Zhou K, Hu S M, Mitra N J. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph., 2012, 31(4): 99:1-99:11.

[27] Chen T, Zhu Z, Shamir A, Hu S M, Cohen-Or D. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics, 2013, 32(6): Article No. 195.

[28] Miao Y, Hu F, Zhang X, Chen J, Pajarola R. SymmSketch: Creating symmetric 3D free-form shapes from 2D sketches. Computational Visual Media, 2015, 1(1): 3-16.

[29] Funkhouser T, Kazhdan M, Shilane P, Min P, KieferW, Tal A, Rusinkiewicz S, Dobkin D. Modeling by example. ACM Trans. Graph., 2004, 23(3): 652-663.

[30] Shin H, Igarashi T. Magic canvas: Interactive design of a 3-D scene prototype from freehand sketches. In Proc. Graphics Interface, May 2007, pp.63-70.

[31] Lee J, Funkhouser T. Sketch-based search and composition of 3D models. In Proc. the 5th SBM, June 2008, pp.97-104.

[32] Xu K, Chen K, Fu H, Sun W L, Hu S M. Sketch2Scene: Sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics, 2013, 32(4): Article No. 123.

[33] Kreavoy V, Julius D, Sheffer A. Model composition from interchangeable components. In Proc. the 15th PG, Oct. 2007, pp.129-138.

[34] Chaudhuri S, Koltun V. Data-driven suggestions for creativity support in 3D modeling. ACM Trans. Graph., 2010, 29(6): 183:1-183:10.

[35] Chaudhuri S, Kalogerakis E, Guibas L, Koltun V. Probabilistic reasoning for assembly-based 3D modeling. ACM Trans. Graph., 2011, 30(4): 35:1-35:10.

[36] Li Y, Sun J, Tang C K, Shum H Y. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303-308.

[37] Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 2008, 77(1/2/3): 157-173.

[38] Barnes C, Shechtman E, Finkelstein A, Goldman D B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24:1-24:11.

[39] Criminisi A, Reid I, Zisserman A. Single view metrology. International Journal of Computer Vision, 2000, 40(2): 123-148.

[40] Sinha S N, Steedly D, Szeliski R, Agrawala M, Pollefeys M. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics, 2008, 27(5): 159:1-159:10.

[41] Wilczkowiak M, Sturm P, Boyer E. Using geometric constraints through parallelepipeds for calibration and 3D modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 194-207.

[42] Jiang N, Tan P, Cheong L F. Symmetric architecture modeling with a single image. ACM Transactions on Graphics, 2009, 28(5): 113:1-113:8.

[43] Shen C H, Fu H, Chen K, Hu S M. Structure recovery by part assembly. ACM Transactions on Graphics, 2012, 31(6): Article No. 180.

[44] O'Donovan P, Agarwala A, Hertzmann A. Color compatibility from large datasets. ACM Transactions on Graphics, 2011, 30(4): Article No. 63.

[45] Yu L F, Yeung S K, Terzopoulos D, Chan T F. DressUp!: Outfit synthesis through automatic optimization. ACM Transactions on Graphics, 2012, 31(6): 134:1-134:14.

[46] Xu K, Zheng H, Zhang H, Cohen-Or D, Liu L, Xiong Y. Photo-inspired model-driven 3D object modeling. ACM Trans. Graph., 2011, 30(4): 80:1-80:10.

[47] Cootes T F, Taylor C J, Cooper D H, Graham J et al. Active shape models — Their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38-59.

[48] Schwarz G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6(2): 461-464.
No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Zhu Mingyuan;. Two Congruent Semantics for Prolog with CUT[J]. , 1990, 5(1): 82 -91 .
[2] Jiang Chanaiun;. Net Operations (Ⅱ)-The Iterated Addition Operation of Petri Nets[J]. , 1995, 10(6): 509 -517 .
[3] CHEN Haiming;. Function Definition Language FDL andIts Implementation[J]. , 1999, 14(4): 414 -421 .
[4] HE Taosong;. Volumetric Virtual Environments[J]. , 2000, 15(1): 37 -46 .
[5] WANG Guoping; HUA Xuanji; SUN Jiaguang;. The Differential Equation Algorithm for General Deformed Swept Volumes[J]. , 2000, 15(6): 604 -610 .
[6] Xiao-Ling Wang, Sheng Huang, and Ao-Ying Zhou. QoS-Aware Composite Services Retrieval[J]. , 2006, 21(4): 547 -558 .
[7] Ke-Yan Cao, Guo-Ren Wang, Dong-Hong Han, Guo-Hui Ding, Ai-Xia Wang, and Ling-Xu Shi. Continuous Outlier Monitoring on Uncertain Data Streams[J]. , 2014, 29(3): 436 -448 .
[8] Peng Liu, Lei Fang, and Michael C. Huang. DEAM：Decoupled, Expressive, Area-Efficient Metadata Cache[J]. , 2014, 29(4): 679 -691 .
[9] Xian Wu, Wei Fan, Jing Gao Zi-Ming Feng, Yong Yu. Detecting Marionette Microblog Users for Improved Information Credibility[J]. , 2015, 30(5): 1082 -1096 .
[10] Shihong Xia, Lin Gao, Yu-Kun Lai, Ming-Ze Yuan, Jinxiang Chai. A Survey on Human Performance Capture and Animation[J]. , 2017, 32(3): 536 -554 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

 Home
Editorial Board
Author Guidelines
Subscription		 Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved