Performance analysis of clustering algorithm under two kinds of big data architecture

Article type: Research Article

Authors: Li, Beibei^a | Liu, Bo^a | Lin, Weiwei^{b; *} | Zhang, Ying^a

Affiliations: [a] School of Computer, South China Normal University, Guangzhou, China. E-mails: 1538339980@qq.com, liugubin@126.com, 2283240387@qq.com | [b] School of Computer Science and Engineering, South China University of Technology, Guangzhou, China. E-mail: linww@scut.edu.cn

Correspondence: [*] Corresponding author. E-mail: linww@scut.edu.cn.

Abstract: To compare the performance of the clustering algorithm on two data processing architectures, the implementations of k-means clustering algorithm on two big data architectures are given at first in this paper. Then we focus on the differences of theoretical performance of k-means algorithm on two architectures from the mathematical point of view. The theoretical analysis shows that Spark architecture is superior to the Hadoop in aspects of the average execution time and I/O time. Finally, a text data set of social networking site of users’ behaviors is employed to conduct algorithm experiments. The results show that Spark is significantly less than MapReduce in aspects of the execution time and I/O time based on k-means algorithm. The theoretical analysis and the implementation technology of the big data algorithm proposed in this paper are a good reference for the application of big data technology.

Keywords: Hadoop, MapReduce, Spark, clustering algorithm, big data, k-means

DOI: 10.3233/JHS-170556

Journal: Journal of High Speed Networks, vol. 23, no. 1, pp. 49-57, 2017