Using microtasks to crowdsource DBpedia entity classification: A study in workflow design

Article type: Research Article

Authors: Bu, Qiong^{; *} | Simperl, Elena | Zerr, Sergej | Li, Yunjia

Affiliations: School of Electronics and Computer Science, University of Southampton, UK. E-mails: qb1g13@soton.ac.uk, e.simperl@soton.ac.uk, s.zerr@soton.ac.uk, yl2@soton.ac.uk

Correspondence: [*] Corresponding author. E-mail: qb1g13@soton.ac.uk.

Abstract: DBpedia is at the core of the Linked Open Data Cloud and widely used in research and applications. However, it is far from being perfect. Its content suffers from many flaws, as a result of factual errors inherited from Wikipedia or incomplete mappings from Wikipedia infobox to DBpedia ontology. In this work we focus on one class of such problems, un-typed entities. We propose a hierarchical tree-based approach to categorize DBpedia entities according to the DBpedia ontology using human computation and paid microtasks. We analyse the main dimensions of the crowdsourcing exercise in depth in order to come up with suggestions for workflow design and study three different workflows with automatic and hybrid prediction mechanisms to select possible candidates for the most specific category from the DBpedia ontology. To test our approach, we run experiments on CrowdFlower using a gold standard dataset of 120 previously unclassified entities. In our studies human-computation driven approaches generally achieved higher precision at lower cost when compared to workflows with automatic predictors. However, each of the tested workflows has its merit and none of them seems to perform exceptionally well on the entities that the DBpedia Extraction Framework fails to classify. We discuss these findings and their potential implications for the design of effective crowdsourced entity classification in DBpedia and beyond.

Keywords: Task design, workflow design, entity classification, DBpedia, microtask crowdsourcing, CrowdFlower

DOI: 10.3233/SW-170261

Journal: Semantic Web, vol. 9, no. 3, pp. 337-354, 2018