Affiliations: [a] College of Information, Shanghai Ocean University, Shanghai, China | [b] School of Computer Engineering and Science, Shanghai University, Shanghai, China | [c] College of Information Technology, Shanghai JianQiao University, Shanghai, China
Correspondence:
[*]
Corresponding author: Guie Jiao, College of Information Technology, Shanghai JianQiao University, Shanghai, China. E-mail: jiaoguie@gench.edu.cn.
Abstract: With the rapid growth of massive data in all walks of life, massive data faces enormous challenges such as storage capacity and computing power. In Chinese universities, traditional data analysis of student course cannot meet the growing demand for increasing data size and real-time computation of big data. In this paper, a parallel FP-Growth algorithm based on split is proposed. The established FP-Tree is split into blocks, and the split FP-Trees are equally divided into different nodes. The monitoring point is set up to monitor the operation of other nodes, dynamically migrate tasks and maintain load balancing. The experiment proves that each node has good load balancing with the given support degree, and the improved algorithm has better running performance than the classic FP-Growth algorithm in parallel processing. Finally, the parallel FP-Growth algorithm based on split is implemented on Hadoop to mine association rules between course grades. The mining process includes data preprocessing, mining results and analysis. The association rules between course grades provide suggestions for the way students learn and the way teachers teach.
Keywords: Association rules, Hadoop, MapReduce, parallel FP-Growth algorithm
