Applying interrater reliability measure for user credibility assessment in reputation-oriented service discovery

Article type: Research Article

Authors: Paul, Arnab | Dhar, Sourish^{; *} | Roy, Sudipta

Affiliations: Department of Computer Science & Engineering, Assam University, Silchar, Assam, India

Correspondence: [*] Corresponding author. E-mail: sourish.dhar@aus.ac.in.

Abstract: Service Oriented Architecture is built on services in which web service discovery is one of the most widely explored domains. Service reputation plays a vital role in the discovery process while selecting the optimal service from a large pool of functionally equivalent services. Reputation mechanism attempts to forecast the future performance of a service based on its past behaviors generally obtained in the form of feedback ratings submitted by users of the service. These feedback ratings about a service may vary from user to user. Variations in feedback ratings could be because of different users’ different subjective judgments and/or dishonest users’ purposely submitted unfair ratings. This paper proposes a service reputation measurement approach counting such diversified act of rating. To realize this goal, an efficient user credibility assessment methodology has been devised by employing the measure of interrater reliability. Experiments are performed to validate the feasibility and effectiveness of the proposed reputation measurement approach. The experimental results depict that the proposed approach can fairly assess service reputations in the presence of various kinds of raters in the system.

Keywords: Service reputation, user credibility, subjective feedback rating, malicious feedback rating, interrater reliability, Cohen’s weighted Kappa, Spearman’s rank-order correlation coefficient

DOI: 10.3233/WEB-220002

Journal: Web Intelligence, vol. 21, no. 2, pp. 167-180, 2023