A protocol for testing expert-system reliability

Article type: Research Article

Authors: François, P.^a | Robert, C.^b | Weil, G.^b | Demongeot, J.^b

Affiliations: [a] Département de pédiatrie, CHU de Grenoble, 38700 La Tronche, France | [b] Service d'information et informatique médicale, CHU de Grenoble, France

Abstract: We set out a protocol for giving a computerised medical decision system the label of expert-system meaning that its responses are of the same standard as those of an expert. We apply it to the validation of an expert-system, MENINGE, which is applied to diagnosis of meningitis in children. The proposed protocol can be widely used since it doesn't refer to the existence of correct responses for therapy or diagnosis, and since it involves simple statistical computations. First this validation rests on a comparison of agreement among experts and among physicians new to medical practice. Secondly, the agreement of the system with the experts is compared to the agreement of the system with physicians new to medical practice. In order to perform this comparison we recommend a standard agreement measure, the choice of which is an important issue in this paper.

Keywords: Agreement, Kappa-coefficient, Entropy, Convex functions, Expert-system

DOI: 10.3233/THC-1994-2102

Journal: Technology and Health Care, vol. 2, no. 1, pp. 19-26, 1994