›› 2015, Vol. 30 ›› Issue (6): 1215-1232.doi: 10.1007/s11390-015-1595-1

Special Issue: Artificial Intelligence and Pattern Recognition

• Special Section on Networking and Distributed Computing for Big Data • Previous Articles     Next Articles

SEIP: System for Efficient Image Processing on Distributed Platform

Tao Liu(刘弢), Yi Liu(刘轶), Member, CCF, Qin Li(李钦), Xiang-Rong Wang(王香荣), Fei Gao(高飞), Yan-Chao Zhu(朱延超), De-Pei Qian(钱德沛), Fellow, CCF   

  1. School of Computer Science and Engineering, Beihang University, Beijing 100191, China
  • Received:2015-05-15 Revised:2015-10-13 Online:2015-11-05 Published:2015-11-05
  • About author:Tao Liu received his B.E. and M.S. degrees in computer science and technology from Shandong University, Jinan, in 2007 and 2010, respectively. Currently, he is a Ph.D. candidate in the School of Computer Science and Engineering, Beihang University, Beijing. He is a member of Sino-German Joint Software Institute at Beihang University. His research interests include parallel computing and high performance computing.
  • Supported by:

    The work was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 61133004, the National High Technology Research and Development 863 Program of China under Grant No. 2012AA01A302, and the NSFC Projects of International Cooperation and Exchanges under Grant No. 61361126011.

Nowadays, there exist numerous images in the Internet, and with the development of cloud computing and big data applications, many of those images need to be processed for different kinds of applications by using specific image processing algorithms. Meanwhile, there already exist many kinds of image processing algorithms and their variations, while new algorithms are still emerging. Consequently, an ongoing problem is how to improve the efficiency of massive image processing and support the integration of existing implementations of image processing algorithms into the systems. This paper proposes a distributed image processing system named SEIP, which is built on Hadoop, and employs extensible innode architecture to support various kinds of image processing algorithms on distributed platforms with GPU accelerators. The system also uses a pipeline-based framework to accelerate massive image file processing. A demonstration application for image feature extraction is designed. The system is evaluated in a small-scale Hadoop cluster with GPU accelerators, and the experimental results show the usability and efficiency of SEIP.

[1] Tanenbaum A S, Van Steen M. Distributed Systems: Principles and Paradigms. Upper Saddle River, NJ: Prentice Hall, 2007, pp.7-8.

[2] Fleischmann A. Distributed Systems: Software Design and Implementation. Springer-Verlag Berlin Heidelberg, 2012, pp.4-5.

[3] Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters. Communications of the ACM, 2008, 51(1): 107-113.

[4] Zaharia M, Chowdhury M, Franklin M J et al. Spark: Cluster computing with working sets. In Proc. the 2nd USENIX Conference on Hot Topics in Cloud Computing, Jun. 2010.

[5] White T. Hadoop: The Definitive Guide (1st edition). O'Reilly Media, Jun. 2009.

[6] Zaharia M, Chowdhury M, Das T et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proc. the 9th USENIX Conference on Networked Systems Design and Implementation, Apr. 2012, pp.15-28.

[7] Ojala T, Pietikainen M, Harwood D. Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In Proc. the 12th International Conference on Pattern Recognition (ICPR), Oct. 1994, Volume 1, pp.582-585.

[8] Ojala T, Pietikainen M, Mäenpää T. Multiresolution grayscale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 971-987.

[9] Bay H, Tuytelaars T, Van Gool L. SURF: Speeded-up robust features. In Proc. the 9th ECCV, May 2006, pp.404- 417.

[10] Ng P C, Henikoff S. SIFT: Predicting amino acid changes that affect protein function. Nucleic Acids Research, 2003, 31(13): 3812-3814.

[11] Tola E, Lepetit V, Fua P. DAISY: An efficient dense descriptor applied to wide-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(5): 815-830.

[12] Juan L, Gwun O. A comparison of SIFT, PCA-SIFT and SURF. International Journal of Image Processing (IJIP), 2009, 3(4): 143-152.

[13] Lewis J, Alghamdi M, Assaf M A et al. An automatic prefetching and caching system. In Proc. the 29th IEEE International on Performance Computing and Communications Conference (IPCCC), Dec. 2010, pp.180-187.

[14] Shvachko K, Kuang H, Radia S et al. The Hadoop distributed file system. In Proc. the 26th IEEE Symposium on Mass Storage Systems and Technologies (MSST), May 2010.

[15] Lindholm E, Nickolls J, Oberman S et al. NVIDIA Tesla: A unified graphics and computing architecture. IEEE Micro, 2008, 28(2): 39-55.

[16] Hartley T D R, Catalyurek U V, Ruiz A et al. Author's retrospective for biomedical image analysis on a cooperative cluster of gpus and multicores. In Proc. the 25th ACM International Conference on Supercomputing Anniversary Volume, Jun. 2014, pp.82-84.

[17] McGaffin M G, Fessler J. Edge-preserving image denoising via group coordinate descent on the GPU. IEEE Transactions on Image Processing, 2015, 24(4): 1273-1281.

[18] Zhu L, Jin H, Zheng R et al. Effective naive Bayes nearest neighbor based image classification on GPU. Journal of Supercomputing, 2014, 68(2): 820-848.

[19] Cornelis N, van Gool L. Fast scale invariant feature detection and matching on programmable graphics hardware. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2008, pp.1-8.

[20] Wu C. SiftGPU: A GPU implementation of scale invariant feature transform (SIFT). http://cs.unc.edu/ ccwu/siftgpu, Oct. 2015.

[21] Prisacariu V, Reid I. fastHOG — A real-time GPU implementation of HOG. Technical Report 2310/09, Department of Engineering Science, University of Oxford, January 2012.

[22] Jiang D, Chen G, Ooi B C et al. epiC: An extensible and scalable system for processing big data. Proceedings of the VLDB Endowment, 2014, 7(7): 541-552.

[23] Zhang X, Yang L T, Liu C et al. A scalable two-phase topdown specialization approach for data anonymization using MapReduce on cloud. IEEE Transactions on Parallel and Distributed Systems, 2014, 25(2): 363-373.

[24] Ranger C, Raghuraman R, Penmetsa A et al. Evaluating MapReduce for multi-core and multiprocessor systems. In Proc. the 13th IEEE International Symposium on High Performance Computer Architecture (HPCA), Feb. 2007, pp.13-24.

[25] Moise D, Shestakov D, Gudmundsson G et al. Terabytescale image similarity search: Experience and best practice. In Proc. IEEE International Conference on Big Data, Oct. 2013, pp.674-682.

[26] Mills S, Eyers D, Leung K C et al. Large-scale feature matching with distributed and heterogeneous computing. In Proc. the 28th IEEE International Conference of Image and Vision Computing New Zealand (IVCNZ), Nov. 2013, pp.208-213.

[27] Teodoro G, Kurç T M, Pan T et al. Accelerating large scale image analyses on parallel, CPU-GPU equipped systems. In Proc. the 26th IEEE International on Parallel and Distributed Processing Symposium (IPDPS), May 2012, pp.1093-1104.

[28] Teodoro G, Pan T F, Kurç T M et al. High-throughput analysis of large microscopy image datasets on CPU-GPU cluster platforms. In Proc. the 27th IEEE International on Parallel and Distributed Processing Symposium (IPDPS), May 2013, pp.103-114.

[29] Hua Y, Jiang H, Feng D. FAST: Near real-time searchable data analytics for the cloud. In Proc. the International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Nov. 2014, pp.754-765.

[30] Liu J, Huang Z, Cheng H et al. Presenting diverse location views with real-time near-duplicate photo elimination. In Proc. the 29th IEEE International Conference on Data Engineering (ICDE), Apr. 2013, pp.505-516.

[31] Fang W, He B, Luo Q et al. Mars: Accelerating MapReduce with graphics processors. IEEE Transactions on Parallel and Distributed Systems, 2011, 22(4): 608-620.

[32] Hong C, Chen D, Chen W et al. MapCG: Writing parallel program portable between CPU and GPU. In Proc. the 19th ACM International Conference on Parallel Architectures and Compilation Techniques (PACT), Sept. 2010, pp.217- 226.

[33] Zhai Y, Mbarushimana E, Li W et al. Lit: A high performance massive data computing framework based on CPU/GPU cluster. In Proc. IEEE International Conference on Cluster Computing (CLUSTER), Sept. 2013.

[34] Jiang H, Chen Y, Qiao Z et al. Accelerating MapReduce framework on multi-GPU systems. Cluster Computing, 2014, 17(2): 293-301.

[35] Jiang H, Chen Y, Qiao Z et al. Scaling up MapReducebased big data processing on multi-GPU systems. Cluster Computing, 2015, 18(1): 369-383.

[36] Wittek P, Darányi S N. Accelerating text mining workloads in a MapReduce-based distributed GPU environment. Journal of Parallel and Distributed Computing, 2013, 73(2): 198-206.
No related articles found!
Full text



[1] Zhang Bo; Zhang Ling;. Statistical Heuristic Search[J]. , 1987, 2(1): 1 -11 .
[2] Meng Liming; Xu Xiaofei; Chang Huiyou; Chen Guangxi; Hu Mingzeng; Li Sheng;. A Tree-Structured Database Machine for Large Relational Database Systems[J]. , 1987, 2(4): 265 -275 .
[3] Lin Qi; Xia Peisu;. The Design and Implementation of a Very Fast Experimental Pipelining Computer[J]. , 1988, 3(1): 1 -6 .
[4] Sun Chengzheng; Tzu Yungui;. A New Method for Describing the AND-OR-Parallel Execution of Logic Programs[J]. , 1988, 3(2): 102 -112 .
[5] Zhang Bo; Zhang Tian; Zhang Jianwei; Zhang Ling;. Motion Planning for Robots with Topological Dimension Reduction Method[J]. , 1990, 5(1): 1 -16 .
[6] Wang Dingxing; Zheng Weimin; Du Xiaoli; Guo Yike;. On the Execution Mechanisms of Parallel Graph Reduction[J]. , 1990, 5(4): 333 -346 .
[7] Zhou Quan; Wei Daozheng;. A Complete Critical Path Algorithm for Test Generation of Combinational Circuits[J]. , 1991, 6(1): 74 -82 .
[8] Zhao Jinghai; Liu Shenquan;. An Environment for Rapid Prototyping of Interactive Systems[J]. , 1991, 6(2): 135 -144 .
[9] Shang Lujun; Xu Lihui;. Notes on the Design of an Integrated Object-Oriented DBMS Family[J]. , 1991, 6(4): 389 -394 .
[10] Xu Jianguo; Gou Yuchai; Lin Zongkai;. HEPAPS:A PCB Automatic Placement System[J]. , 1992, 7(1): 39 -46 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
E-mail: jcst@ict.ac.cn
  Copyright ©2015 JCST, All Rights Reserved