Affiliations: [a] Change Management Tool S.L., Barcelona, Spain | [b] Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain | [c] Universitat Autònoma de Barcelona, Bellaterra, Spain. ewa@iiia.csic.es | [d] Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain. jar@iiia.csic.es | [e] Artificial Intelligence Research Institute (IIIA-CSIC), Bellaterra, Spain. sierra@iiia.csic.es
Correspondence:
[*]
Address for correspondence: Campus UAB, 08193 Bellaterra, Spain. Supported by an Industrial PhD scholarship from the Generalitat de Catalunya (DI-060).
Abstract: In this paper we introduce a new ranking algorithm, called Collaborative Judgement (CJ), that takes into account peer opinions of agents and/or humans on objects (e.g. products, exams, papers) as well as peer judgements over those opinions. The combination of these two types of information has not been studied in previous work in order to produce object rankings. Here we apply Collaborative Judgement to the use case of scientific paper assessment and we validate it over simulated data. The results show that the rankings produced by our algorithm improve current scientific paper ranking practice, which is based on averages of opinions weighted by their reviewers’ self-assessments.
