A new architecture for improving focused crawling using deep neural network

Article type: Research Article

Authors: ElAraby, M.E.^{a; b; *} | Abuelenin, Sherihan M.^a | Moftah, Hossam M.^b | Rashad, M.Z.^a

Affiliations: [a] Faculty of Computers and Information, Mansoura University, Egypt | [b] Faculty of Computers and Information, Beni-Suef University, Egypt

Correspondence: [*] Corresponding author. M.E. ElAraby, Faculty of Computers and Information, Mansoura University, Egypt. E-mail: mohamed.elaraby@fcis.bsu.edu.eg.

Abstract: Cloud computing offers elastic features to alleviate the challenges of web crawling. Building crawlers in a scalable fashion has become highly needed. This paper proposes a new Focused Crawler (FC) architecture that can be introduced as a service over the cloud computing. The proposed FC has a service called a Topic Filter Service (TFS), which is responsible for filtering retrieved pages before indexing and extracting links to add them in the crawling queue. TFS relies on the Deep Neural Network (DNN) classifier. TFS is trained by a dataset. This dataset is processed by an outlier rejection using support vector machine classifier. Moreover, this proposed FC has a further service called Concept Weight Handler (CWH). It is responsible for handling the keywords such as concepts based on meanings and it calculates the weight of each concept. Experimental results show that cloud computing services provide a better environment for running and improving the speed of crawling. The proposed classifier has been tested in comparison with other classification techniques and has proved highly accurate. The overall accuracy offered by the employed architecture confirms that the effectiveness and performance of the proposed FC is high.

Keywords: Focused Crawler, deep neural network, cloud computing, topic filter service, concept web page

DOI: 10.3233/JIFS-182683

Journal: Journal of Intelligent & Fuzzy Systems, vol. 37, no. 1, pp. 1233-1245, 2019