Abstract In this paper, we present a practical system to automatically suggest the most pairing clothing items, given the reference clothing (upper-body or low-body). However this has being a challenge, due to having varieties of clothing categories. Clothing is one of the most informative cues for human appearance. In our daily life, people need to wear properly and beautifully to show their confidence, politeness and social status in various occasion. But, it is not easy to decide to decide on what and how to wear at the same time to match with the selected clothes. To address this problem, we propose a quadruple network architecture, where one dual network adopts Siamese convolution neural network architecture. Training examples are pairs of upper-body and low-body clothing items that are either compatible or incompatible. Another dual convolution neural network is used to learn clothing style features of the input image. This framework allows learning a feature transformation from the images of clothing items to two latent spaces, which we called compatible space and style space. After training the two dual networks, we use a distance fusion method to fuse the features extracted from the compatible and style dual networks. To acquire an optimized model and verify our proposed method, we expand a large clothing dataset called "How to Wear Beautifully" (H2WB). Experiments on H2WB dataset demonstrated that our learning model are effective with feature distance fusion and clothing item pairing recommendation.
This work was supported by the National Natural Science Foundation of China under Grant Nos. 61379106, 61379082 and 61227802, the Natural Science Foundation of Shandong Province of China under Grant Nos. ZR2013FM036 and ZR2015FM011, the Major Program of National Natural Science Foundation of China under Grant No. 41631073, and the Qingdao National Laboratory for Marine Science and Technology of China under Grant No. QNLM2016ORP0401.
About author: Wen-Guang Chen is a professor in the Department of Computer Science and Technology, Tsinghua University, Beijing, where he has been teaching since 2003. He received his B.S. and Ph.D. degrees both in computer science from Tsinghua University in 1995 and 2000 respectively. His research interest is in parallel and distributed computing. He is a CCF distinguished member and a CCF distinguished speaker, and an ACM member and the vice chair of ACM China Council. He has served in program committees of a variety of major conferences in the parallel and distributed computing area, including PLDI, PPoPP, SC, CGO, CCGrid, IPDPS, APSys, and ICPP. He is the editor-in-chief of Communication of ACM China Edition.
Cite this article:
Yu-Jie Liu, Yong-Biao Gao, Ling-Yan Bian, Wen-Ya Wang, Zong-Min Li.How to wear beautiful? Clothing pair recommendation[J] Journal of Computer Science and Technology, 2018,V33(3): 522-530
 Kiapour M H, Han X, Lazebnik S, Berg A C. Where to buy it:Matching street clothing photos in online shops. In Proc. IEEE International Conference on Computer Vision, Dec. 2015, pp.3343-3351. Li Z, Li Y, Gao Y, Liu Y. Fast cross-scenario clothing retrieval based on indexing deep features. In Proc. PacificRim Conference on Advances in Multimedia Information Processing, Sept. 2016, pp.107-118. Lu H, Xu C, Liu G, Song Z, Liu S, Yan S. Street-to-shop:Cross-scenario clothing retrieval via parts alignment and auxiliary set. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2012, pp.1335-1336. Chen Q, Huang J, Feris R, Brown L M, Dong J, Yan S. Deep domain adaptation for describing people based on fine-grained clothing attributes. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.5315-5324. Liu Z, Luo P, Qiu S, Wang X, Tang X. DeepFashion:Powering robust clothes recognition and retrieval with rich annotations. In Proc. IEEE International Conference on Computer Vision and Pattern Recognition, June 2016, pp.1096-1104. Zhang X, Jia J, Gao K, Zhang Y, Zhang D, Li J, Tian Q. Trip outfits advisor:Location-oriented clothing recommendation. IEEE Transactions on Multimedia, 2017, 19(11):2533-2544. Liu S, Nguyen T V, Feng J, Wang M, Yan S. Hi, magic closet, tell me what to wear! In Proc. ACM International Conference on Multimedia, October 2012, pp.1333-1334. Liu Y, Gao Y, Feng S, Li Z. Weather-to-garment:Weatheroriented clothing recommendation. In Proc. IEEE International Conference on Multimedia and Expo, June 2017, pp.181-186. Sha D, Wang D, Zhou X, Feng S, Zhang Y, Yu G. An Approach for Clothing Recommendation Based on Multiple Image Attributes. Springer International Publishing, 2016. Cheng M M, Hou Q B, Zhang S H et al. Intelligent visual media processing:When graphics meets vision. Journal of Computer Science and Technology, 2017, 32(1):110-121. Simo-Serra E, Fidler S, Moreno-Noguer F, Urtasun R. Neuroaesthetics in fashion:Modeling the perception of fashionability. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.869-877. Jagadeesh V, Piramuthu R, Bhardwaj A, Di W, Sundaresan N. Large scale visual recommendations from street fashion images. In Proc. ACM SIGKDD international conference on Knowledge discovery and data mining, August 2014, pp.1925-1934. Yamaguchi K, Berg T L, Ortiz L E. Chic or social:Visual popularity analysis in online fashion networks. In Proc. ACM International Conference on Multimedia, Nov. 2014, pp.773-776. Zheng Z H, Zhang H T, Zhang F L, Mu T J. Imagebased clothes changing system. Computational Visual Media, 2017, 3(4):337-347. Ren S, He K, Girshick R, Sun J. Faster R-CNN:Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6):1137-1149. Di W, Wah C, Bhardwaj A, Piramuthu R, Sundaresan N. Style finder:Fine-grained clothing style detection and retrieval. In Proc. IEEE International Conference on Computer Vision and Pattern Recognition Workshops, June 2013, pp.8-13. Chen K T, Chen K, Cong P, Hsu W H, Luo J. Who are the devils wearing prada in New York city? In Proc. ACM International Conference on Multimedia, August 2015, pp.177-180. Yamaguchi K, Kiapour M H, Ortiz L E, Berg T L. Parsing clothing in fashion photographs. In Proc. IEEE International Conference on Computer Vision and Pattern Recognition, June 2012, pp.3570-3577. Garcia N, Vogiatzis G. Dress like a star:Retrieving fashion products from videos. In Proc. IEEE International Conference on Computer Vision, October 2017. Cheng Z Q, Wu X, Liu Y, Hua X S. Video2Shop:Exact matching clothes in videos to online shopping images. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2017, pp.4169-4177. Kalantidis Y, Kennedy L, Li L J. Getting the look:Clothing recognition and segmentation for automatic product suggestions in everyday photos. In Proc. International Conference on Multimedia Retrieval, April 2013, pp.105-112. Shen E, Lieberman H, Lam F. What am I gonna wear?:Scenario-oriented recommendation. In Proc. International Conference on Intelligent User Interfaces, January 2007, pp.365-368. Zhao Y, Araki K. A recommendation system for a fashion coordination based on user's information. Ieice Technical Report Life Intelligence and Office Information Systems, 2012, 111:13-18. Hu Y, Yi X, Davis L S. Collaborative fashion recommendation:A functional tensor factorization approach. In Proc. ACM International Conference on Multimedia, October 2015, pp.129-138. Veit A, Kovacs B, Bell S, Mcauley J. Learning visual clothing style with heterogeneous dyadic co-occurrences. In Proc. IEEE International Conference on Computer Vision, Dec. 2015, pp.4642-4650. Song X, Feng F, Liu J, Li Z, Nie L, Ma J. Neurostylist:Neural compatibility modeling for clothing matching. In Proc. ACM International Conference on Multimedia, October 2017, pp.753-761. Melekhov I, Kannala J, Rahtu E. Siamese network features for image matching. In Proc. International Conference on Pattern Recognition, April 2017, pp.378-383. Hidayati S C, You C W, Cheng W H, Hua K L. Learning and recognition of clothing genres from full-body images. IEEE Transactions on Cybernetics, 2017, PP(99):1-13. Liu Q, Wu S, Wang L. DeepStyle:Learning user preferences for visual recommendation. In Proc. the International ACM SIGIR Conference, June 2017, pp.841-844. Jia Y, Shelhamer E, Donahue J et al. Caffe:Convolutional architecture for fast feature embedding. In Proc. ACM International Conference on Multimedia, June 2014, pp.675-678. Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural Networks. In Proc. International Conference on Neural Information Processing Systems, January 2012, pp.1097-1105. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3):211-252. Yang Y, Ramanan D. Articulated pose estimation with exible mixtures-of-parts. In Proc. IEEE International Conference on Computer Vision and Pattern Recognition, June 2011, pp.1385-1392. Liang X, Lin L, Yang W, Luo P, Huang J, Yan S. Clothes coparsing via joint image segmentation and labeling with application to clothing retrieval. IEEE Transactions on Multimedia, 2016, 18(6):1175-1186. Hao P Y, Xia Y, Li X X, Kamata S I, Chen S Y. Discriminative histogram intersection metric learning and its applications. Journal of Computer Science and Technology, 2017, 32(3):507-519. Rubner Y, Tomasi C, Guibas L J. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 2000, 40(2):99-121. Siddiquie B, Feris R S, Davis L S. Image ranking and retrieval based on multi-attribute queries. In Proc. IEEE International Conference on Computer Vision and Pattern Recognition, June 2011, pp.801-808.
Copyright 2010 by Journal of Computer Science and Technology