›› 2015, Vol. 30 ›› Issue (3): 499-510.doi: 10.1007/s11390-015-1540-3

Special Issue: Artificial Intelligence and Pattern Recognition; Computer Graphics and Multimedia

• Special Section on Computational Visual Media • Previous Articles     Next Articles

Facial similarity learning with humans in the loop

Chong Cao(曹翀), Student Member, CCF, Hai-Zhou Ai(艾海舟), Senior Member, IEEE, Member, CCF   

  1. Tsinghua National Laboratory for Information Science and Technology, Department of Computer Science and Technology Tsinghua University, Beijing 100084, China
  • Received:2014-11-28 Revised:2015-03-07 Online:2015-05-05 Published:2015-05-05
  • About author:Chong Cao received her B.S. degree in computer science and technology from Tsinghua University, Beijing, in 2010. She is currently a Ph.D. candidate at Tsinghua University. Her research interests include computer vision, pattern recognition and multimedia, with special focus on face retrieval and facial similarity learning.
  • Supported by:

    This work was partly supported by the National Basic Research 973 Program of China under Grant No. 2011CB302203.

Similarity learning has always been a popular topic in computer vision research. Among this, facial similarity is especially important and difficult due to its wide applications and the nonrigid nature of human faces. The large gap between feature representations and human perceptual descriptions makes the problem even harder. In this paper, we learn facial similarity through human-computer interactions. To learn perceptual similarities of faces in a gallery set, we ask users to label some candidate images with their similarities to a probe image. Based on users' responses, a sampling algorithm actively generates a probe image and a set of candidates for the next query. Assisted with human efforts, the algorithm embeds all the images in a space where the distance between two subjects conforms to their dissimilarity in human perception. We apply the learned embedding to face retrieval and compare our method to some feature-based methods on a dataset we collect from social network sites (SNS). Experimental results demonstrate that incorporating human efforts can ensure retrieval accuracy. At the same time, the active sampling algorithm reduces human efforts.

[1] Yuen P C, Man C. Human face image searching system using sketches. IEEE Trans. Systems, Man and Cybernetics, Part A: Systems and Humans, 2007, 37(4): 493-504.

[2] Kumar N, Berg A C, Belhumeur P N, Nayar S K. Attribute and simile classifiers for face verification. In Proc. the 12th IEEE International Conference on Computer Vision (ICCV), Sept. 29-Oct. 2, 2009, pp.365-372.

[3] Jain A K, Vailaya A. Image retrieval using color and shape. Pattern Recognition, 1996, 29(8): 1233-1244.

[4] Ahonen T, Hadid A, Pietikäinen M. Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Analysis and Machine Intelligence, 2006, 28(12): 2037-2041.

[5] Wiskott L, Fellous J M, Krüger N et al. Face recognition by elastic bunch graph matching. IEEE Trans. Pattern Analysis and Machine Intelligence, 1997, 19(7): 775-779.

[6] Berg T, Belhumeur P N. Tom-vs-pete classifiers and identity-preserving alignment for face verification. In Proc. the British Machine Vision Conference, September 2012.

[7] Chopra S, Hadsell R, LeCun Y. Learning a similarity metric discriminatively, with application to face verification. In Proc. the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2005, pp.539-546.

[8] Schroff F, Treibitz T, Kriegman D, Belongie S. Pose, illumination and expression invariant pairwise face-similarity measure via doppelgänger list comparison. In Proc. the 13th IEEE ICCV, Nov. 2011, pp.2494-2501.

[9] Yin Q, Tang X, Sun J. An associate-predict model for face recognition. In Proc. the IEEE CVPR, June 2011, pp.497-504.

[10] Scheirer W J, Kumar N, Belhumeur P N, Boult T E. Multiattribute spaces: Calibration for attribute fusion and similarity search. In Proc. the IEEE CVPR, June 2012, pp.2933-2940.

[11] Siddiquie B, Feris R S, Davis L S. Image ranking and retrieval based on multi-attribute queries. In Proc. the IEEE CVPR, June 2011, pp.801-808.

[12] Parikh D, Grauman K. Relative attributes. In Proc. the 13th IEEE Int. Conf. Computer Vision, Nov. 2011, pp.503-510.

[13] Biswas A, Parikh D. Simultaneous active learning of classifiers & attributes via relative feedback. In Proc. the IEEE CVPR, June 2013, pp.644-651.

[14] Liang L, Grauman K. Beyond comparing image pairs: Setwise active learning for relative attributes. In Proc. the IEEE CVPR, June 2014, pp.208-215.

[15] Agarwal S, Wills J, Cayton L et al. Generalized non-metric multidimensional scaling. In Proc. the 11th Int. Conf. Artificial Intelligence and Statistics, March 2007, pp.11-18.

[16] Tamuz O, Liu C, Belongie S, Shamir O, Kalai A T. Adaptively learning the crowd kernel. arXiv:1105.1033, 2011. http://arxiv.org/abs/1105.1033, March 2015.

[17] van der Maaten L, Weinberger K. Stochastic triplet embedding. In Proc. IEEE International Workshop on Machine Learning for Signal Processing, September 2012.

[18] Garces E, Agarwala A, Gutierrez D, Hertzmann A. A similarity measure for illustration style. ACM Transactions on Graphics, 2014, 33(4): 93:1-93:9.

[19] Deng J, Krause J, Li F F. Fine-grained crowdsourcing for fine-grained recognition. In Proc. IEEE CVPR, June 2013, pp.580-587.

[20] Holub A, Liu Y H, Perona P. On constructing facial similarity maps. In Proc. IEEE CVPR, June 2007.

[21] Kruskal J B. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 1964, 29(1): 1-27.

[22] Fang Y, Geman D. Experiments in mental face retrieval. In Lecture Notes in Computer Science Volume 3546, Kanade T, Jain A, Ratha N K (eds.), Springer, 2005, pp.637-646.

[23] Ruiz-del-Solar J, Navarrete P. FACERET: An interactive face retrieval system based on self-organizing maps. In Proc. Int. Conf. Image and Video Retrieval, July 2002, pp.157-164.

[24] Yang Z, Laaksonen J. Partial relevance in interactive facial image retrieval. In Proc. the 3rd Int. Conf. Advances in Pattern Recognition, Part II, August 2005, pp.216-225.

[25] Cao C, Kwak S, Belongie S, Kriegman D, Ai H. Adaptive ranking of facial attractiveness. In Proc. the IEEE International Conference on Multimedia and Expo, July 2014.

[26] Huang G B, Ramesh M, Berg T, Learned-Miller E. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst, Oct. 2007.

[27] Huang C, Ai H, Li Y, Lao S. High-performance rotation invariant multiview face detection. IEEE Trans. Pattern Analysis and Machine Intelligence, 2007, 29(4): 671-686.

[28] Yan D, Huang L, Jordan M I. Fast approximate spectral clustering. In Proc. the 15th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, June 2009, pp.907-916.
No related articles found!
Full text



[1] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[2] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[3] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[4] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[5] Huang Heyan;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[6] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[7] Tang Tonggao; Zhao Zhaokeng;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[8] Min Yinghua;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[9] Zhu Hong;. Some Mathematical Properties of the Functional Programming Language FP[J]. , 1987, 2(3): 202 -216 .
[10] Li Minghui;. CAD System of Microprogrammed Digital Systems[J]. , 1987, 2(3): 226 -235 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
E-mail: jcst@ict.ac.cn
  Copyright ©2015 JCST, All Rights Reserved