Affiliations: Software Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran
Correspondence:
[*]
Corresponding author: Kamran Zamanifar, Software Department, Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran. E-mail: Zamanifar@eng.ui.ac.ir.
Abstract: The emergence of Big Data has had a profound impact on how data are analyzed. Open source distributed stream processing platforms have gained popularity for analyzing streaming Big Data as they provide low latency required for streaming Big Data applications using cluster resources. However, existing resource schedulers are still lacking the efficiency that Big Data analytical applications require. Recent works have already considered streaming Big Data characteristics to improve the efficiency of scheduling in the platforms. Nevertheless, they have not taken into account the specific attributes of analytical applications. This study, therefore, presents Bframework, an efficient resource scheduling framework used by streaming Big Data analysis applications based on cluster resources. Bframework proposes a query model using Directed Graphs (DGs) and introduces operator assignment and operator scheduling algorithms based on a novel partitioning algorithm. Bframework is highly adaptable to the fluctuation of streaming Big Data and the availability of cluster resources. Experiments with the benchmark and well-known real-world queries show that Bframework can significantly reduce the latency of streaming Big Data analysis queries up to about 65%.
