Trends in web data extraction using machine learning

Article type: Research Article

Authors: Patnaik, Sudhir Kumar^{; *} | Narendra Babu, C.

Affiliations: Department of Computer Science, MS Ramaiah University of Applied Sciences, MSR Nagar, Bangalore, India. E-mails: skpatnaik9@gmail.com, narendrababu.c@gmail.com

Correspondence: [*] Corresponding author. E-mail: skpatnaik9@gmail.com.

Abstract: Web data extraction has seen significant development in the last decade since its inception in the early nineties. It has evolved from a simple manual way of extracting data from web page and documents to automated extraction to an intelligent extraction using machine learning algorithms, tools and techniques. Data extraction is one of the key components of end-to-end life cycle in web data extraction process that includes navigation, extraction, data enrichment and visualization. This paper presents the journey of web data extraction over the last many years highlighting evolution of tools, techniques, frameworks and algorithms for building intelligent web data extraction systems. The paper also throws light into challenges, opportunities for future research and emerging trends over the years in web data extraction with specific focus on machine learning techniques. Both traditional and machine learning approaches to manual and automated web data extraction are experimented and results published with few use cases demonstrating the challenges in web data extraction in the event of changes in the website layout. This paper introduces novel ideas such as self-healing capability in web data extraction and proactive error detection in the event of changes in website layout as an area of future research. This unique perspective will help readers to get deeper insights in to the present and future of web data extraction.

Keywords: Automated, data quality, machine learning, navigation, web data extraction

DOI: 10.3233/WEB-210465

Journal: Web Intelligence, vol. 19, no. 3, pp. 169-190, 2021