A comparative study of methods for a priori prediction of MCQ difficulty

Issue title: Open Science Data and the Semantic Web Journal

Article type: Research Article

Authors: Kurdi, Ghader^{a; *} | Leo, Jared^a | Matentzoglu, Nicolas^a | Parsia, Bijan^a | Sattler, Uli^a | Forge, Sophie^b | Donato, Gina^b | Dowling, Will^b

Affiliations: [a] Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M139PL, UK | [b] Elsevier, 1600 John F. Kennedy Boulevard, Philadelphia, PA 19103, USA

Correspondence: [*] Corresponding author. E-mail: ghader.kurdi@manchester.ac.uk.

Abstract: Successful exams require a balance of easy, medium, and difficult questions. Question difficulty is generally either estimated by an expert or determined after an exam is taken. The latter provides no utility for the generation of new questions and the former is expensive both in terms of time and cost. Additionally, it is not known whether expert prediction is indeed a good proxy for estimating question difficulty. In this paper, we analyse and compare two ontology-based measures for difficulty prediction of multiple choice questions, as well as comparing each measure with expert prediction (by 15 experts) against the exam performance of 12 residents over a corpus of 231 medical case-based questions that are in multiple choice format. We find one ontology-based measure (relation strength indicativeness) to be of comparable performance (accuracy = 47%) to expert prediction (average accuracy = 49%).

Keywords: Ontologies, semantic web, automatic question generation, difficulty modelling, difficulty prediction, multiple choice questions, student assessment

DOI: 10.3233/SW-200390

Journal: Semantic Web, vol. 12, no. 3, pp. 449-465, 2021