We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 32 Issue 5
September  2017
Turn off MathJax
Article Contents
Abstract
References
Related Articles
Yi-Li Fang, Hai-Long Sun, Peng-Peng Chen, Ting Deng. Improving the Quality of Crowdsourced Image Labeling via Label Similarity[J]. Journal of Computer Science and Technology, 2017, 32(5): 877-889. DOI: 10.1007/s11390-017-1770-7
Citation: Yi-Li Fang, Hai-Long Sun, Peng-Peng Chen, Ting Deng. Improving the Quality of Crowdsourced Image Labeling via Label Similarity[J]. Journal of Computer Science and Technology, 2017, 32(5): 877-889. DOI: 10.1007/s11390-017-1770-7
shu

Improving the Quality of Crowdsourced Image Labeling via Label Similarity

  • 1 State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China;
    2 School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Funds: This work was supported partly by the National Key Research and Development Program of China under Grant No. 2016YFB1000804, the National Natural Science Foundation of China under Grant No. 61602023, the National Basic Research 973 Program of China under Grant Nos. 2014CB340304 and 2015CB358700, and the State Key Laboratory of Software Development Environment under Grant No. SKLSDE-2017ZX-14.
More Information
  • Author Bio:

    Yi-Li Fang is a Ph.D. student in the School of Computer Science and Engineering, Beihang University, Beijing. His research interests mainly include crowd computing/crowdsourcing, social computing, and decision science.

  • Corresponding author:

    Hai-Long Sun,sunhl@act.buaa.edu.cn

  • Received Date: March 01, 2017
  • Revised Date: July 10, 2017
  • Published Date: September 04, 2017
    Abstract
  • Crowdsourcing is an effective method to obtain large databases of manually-labeled images, which is especially important for image understanding with supervised machine learning algorithms. However, for several kinds of tasks regarding image labeling, e.g. dog breed recognition, it is hard to achieve high-quality results. Therefore, further optimizing crowdsourcing workflow mainly involves task allocation and result inference. In task allocation, we design a two-round crowdsourcing framework, which contains a smart decision mechanism based on information entropy to determine whether to perform a second round task allocation. Regarding result inference, after quantifying the similarity of all labels,two graphical models are proposed to describe the labeling process and corresponding inference algorithms are designed to further improve the result quality of image labeling. Extensive experiments on real-world tasks in Crowdflower and synthesis datasets were conducted. The experimental results demonstrate the superiority of these approaches in comparison with state-of-the-art methods.
    • FullText(HTML)
    • References (30)
  • [1]
    Whitehill J, Ruvolo P, Wu T, Bergsma J, Movellan J R. Whose vote should count more:Optimal integration of labels from labelers of unknown expertise. In Proc. the 23rd Annual Conference on Neural Information Processing Systems (NIPS), December 2009, pp.2035-2043.
    [2]
    Li G, Wang J, Zheng Y, Franklin M J. Crowdsourced data management:A survey. IEEE Trans. Knowl. Data Eng., 2016, 28(9):2296-2319.
    [3]
    Deng J, Dong W, Socher R, Li L, Li K, Li F. ImageNet:A large-scale hierarchical image database. In Proc. the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2009, pp.248-255.
    [4]
    von Ahn L, Dabbish L. Labeling images with a computer game. In Proc. the 2004 Conference on Human Factors in Computing Systems (CHI), April 2004, pp.319-326.
    [5]
    Han T, Sun H, Song Y, Fang Y, Liu X. Incorporating external knowledge into crowd intelligence for more specific knowledge acquisition. In Proc. the 25th International Joint Conference on Artificial Intelligence (IJCAI), July 2016, pp.1541-1547.
    [6]
    Fang Y, Sun H, Li G, Zhang R, Huai J. Effective result inference for context-sensitive tasks in crowdsourcing. In Proc. the 21st Database Systems for Advanced Applications (DASFAA), April 2016, pp.33-48.
    [7]
    Li G, Zheng Y, Fan J, Wang J, Cheng R. Crowdsourced data management:Overview and challenges. In Proc. the ACM International Conference on Management of Data (SIGMOD), May 2017, pp.1711-1716.
    [8]
    Li G, Chai C, Fan J, Weng X, Li J, Zheng Y, Li Y, Yu X, Zhang X, Yuan H. CDB:Optimizing queries with crowd-based selections and joins. In Proc. the ACM International Conference on Management of Data (SIGMOD), May 2017, pp.1463-1478.
    [9]
    Hu H, Zheng Y, Bao Z, Li G, Feng J, Cheng R. Crowdsourced POI labelling:Location-aware result inference and task assignment. In Proc. the 32nd IEEE International Conference on Data Engineering (ICDE), May 2016, pp.61-72.
    [10]
    Zheng Y, Wang J, Li G, Cheng R, Feng J. QASCA:A quality-aware task assignment system for crowdsourcing applications. In Proc. the ACM SIGMOD International Conference on Management of Data (SIGMOD), May 31-June 4, 2015, pp.1031-1046.
    [11]
    Zheng Y, Cheng R, Maniu S, Mo L. On optimality of jury selection in crowdsourcing. In Proc. the 18th International Conference on Extending Database Technology (EDBT), Mar. 2015, pp.193-204.
    [12]
    Zheng Y, Li G, Li Y, Shan C, Cheng R. Truth inference in crowdsourcing:Is the problem solved? Proceedings of the VLDB Endowment (PVLDB), 2017, 10(5):541-552.
    [13]
    Zheng Y, Li G, Cheng R. DOCS:A domain-aware crowdsourcing system using knowledge bases. Proceedings of the VLDB Endowment (PVLDB), 2016, 10(4):361-372.
    [14]
    von Ahn L. Duolingo:Learn a language for free while helping to translate the Web. In Proc. the 18th International Conference on Intelligent User Interfaces (IUI), March 2013.
    [15]
    Hu J, Oh J, Gershman A. Learning lexical entries for robotic commands using crowdsourcing. arXiv:1609.02549, 2016. https://arxiv.org/abs/1609.02549, July 2017.
    [16]
    Li Z, Wang T, Zhang Y, Zhan Y, Yin G. Query reformulation by leveraging crowd wisdom for scenario-based software search. In Proc. the 8th Asia-Pacific Symposium on Internetware, September 2016, pp.36-44.
    [17]
    Dawid A P, Skene A M. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, 1979, 28(1):20-28.
    [18]
    Raykar V C, Yu S, Zhao L H, Valadez G H, Florin C, Bogoni L, Moy L. Learning from crowds. J. Mach. Learn. Res., 2010, 11:1297-1322.
    [19]
    Lakshminarayanan B, Teh Y W. Inferring ground truth from multi-annotator ordinal data:A probabilistic approach. arXiv:1305.0015, 2013. https://arxiv.org/abs/1305.0015, July 2017.
    [20]
    Mamykina L, Smyth T N, Dimond J P, Gajos K Z. Learning from the crowd:Observational learning in crowdsourcing communities. In Proc. CHI Conference on Human Factors in Computing Systems (CHI), May 2016, pp.2635-2644.
    [21]
    Pan S, Larson K, Bradshaw J, Law E. Dynamic task allocation algorithm for hiring workers that learn. In Proc. the 25th International Joint Conference on Artificial Intelligence, July 2016, pp.3825-3831.
    [22]
    Liu Y, Liu Y, Zhang M, Ma S. Pay me and I'll follow you:Detection of crowdturfing following activities in microblog environment. In Proc. the 25th International Joint Conference on Artificial Intelligence (IJCAI), July 2016, pp.3789-3796.
    [23]
    Liu Q, Peng J, Ihler A T. Variational inference for crowdsourcing. In Proc. the 26th Advances in Neural Information Processing Systems (NIPS), December 2012, pp.701-709.
    [24]
    Sheng V S, Provost F J, Ipeirotis P G. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proc. the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), August 2008, pp.614-622.
    [25]
    Gao C, Lu Y, Zhou D. Exact exponent in optimal rates for crowdsourcing. In Proc. the 33rd International Conference on Machine Learning (ICML), June 2016, pp.603-611.
    [26]
    Zhang Y, Chen X, Zhou D, Jordan M I. Spectral methods meet EM:A provably optimal algorithm for crowdsourcing. In Proc. the 28th Advances in Neural Information Processing Systems (NIPS), December 2014, pp.1260-1268.
    [27]
    Zhou D, Platt J C, Basu S, Mao Y. Learning from the wisdom of crowds by minimax entropy. In Proc. the 26th Advances in Neural Information Processing Systems (NIPS), December 2012, pp.2204-2212.
    [28]
    Chiang L H, Russell R D. Pattern Classification. Springer London, 2001.
    [29]
    Yang J, Bozzon A, Houben G. Knowledge crowdsourcing acceleration. In Proc. the 15th International Conference Engineering the Web in the Big Data Era, June 2015, pp.639-643.
    [30]
    Li H, Zhao B, Fuxman A. The wisdom of minority:Discovering and targeting the right group of workers for crowdsourcing. In Proc. the 23rd International Conference on World Wide Web (WWW), April 2014, pp.165-176.
    • Relative Articles

  • Related Articles

    [1]Shi-Yao Cui, Bo-Wen Yu, Xin Cong, Ting-Wen Liu, Qing-Feng Tan, Jin-Qiao Shi. Label-Aware Chinese Event Detection with Heterogeneous Graph Attention Network[J]. Journal of Computer Science and Technology, 2024, 39(1): 227-242. DOI: 10.1007/s11390-023-1541-6
    [2]Min Shi, Hao Lu, Zhao-Xin Li, Deng-Ming Zhu, Zhao-Qi Wang. Accurate Robotic Grasp Detection with Angular Label Smoothing[J]. Journal of Computer Science and Technology, 2023, 38(5): 1149-1161. DOI: 10.1007/s11390-022-1458-5
    [3]Peng Ni, Su-Yun Zhao, Zhi-Gang Dai, Hong Chen, Cui-Ping Li. Partial Label Learning via Conditional-Label-Aware Disambiguation[J]. Journal of Computer Science and Technology, 2021, 36(3): 590-605. DOI: 10.1007/s11390-021-0992-x
    [4]Xiu-Yi Jia, Sai-Sai Zhu, Wei-Wei Li. Joint Label-Specific Features and Correlation Information for Multi-Label Learning[J]. Journal of Computer Science and Technology, 2020, 35(2): 247-258. DOI: 10.1007/s11390-020-9900-z
    [5]Peng-Peng Chen, Hai-Long Sun, Yi-Li Fang, Jin-Peng Huai. Collusion-Proof Result Inference in Crowdsourcing[J]. Journal of Computer Science and Technology, 2018, 33(2): 351-365. DOI: 10.1007/s11390-018-1823-6
    [6]Li Zhang, Xin-Yue Huang, Jing Jiang, Ya-Kun Hu. CSLabel:An Approach for Labelling Mobile App Reviews[J]. Journal of Computer Science and Technology, 2017, 32(6): 1076-1089. DOI: 10.1007/s11390-017-1784-1
    [7]Yun-Gang Bao, Sa Wang. Labeled von Neumann Architecture for Software-Defined Cloud[J]. Journal of Computer Science and Technology, 2017, 32(2): 219-223. DOI: 10.1007/s11390-017-1716-0
    [8]Meng Chen, Xiaohui Yu, Yang Liu. Mining Object Similarity for Predicting Next Locations[J]. Journal of Computer Science and Technology, 2016, 31(4): 649-660. DOI: 10.1007/s11390-016-1654-2
    [9]Chong Cao, Hai-Zhou Ai. Facial similarity learning with humans in the loop[J]. Journal of Computer Science and Technology, 2015, 30(3): 499-510. DOI: 10.1007/s11390-015-1540-3
    [10]Ying-Jun Wu, Han Huang, Zhi-Feng Hao, Feng Chen. Local Community Detection Using Link Similarity[J]. Journal of Computer Science and Technology, 2012, 27(6): 1261-1268. DOI: 10.1007/s11390-012-1302-4
    • Supplements (0)

Catalog

    Get Citation
    PDF
    XML
    Article views (40) PDF downloads (975) Cited by()
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return