A conceptual model for ontology quality assessment

1.Introduction

An ontology is a formal representation of the concepts in a particular domain of knowledge and the relationships between those concepts [61]. More specifically, an ontology has been defined as “a formal, explicit specification of a shared conceptualization” in [123] by merging the two prominent definitions of Gruber [61]: “ontology is an explicit specification of a conceptualization” and Borster [21]: “an ontology is a formal specification of a shared conceptualization”. The conceptualization denotes an abstract and simplified view of the world that consists of concepts that are expected in the world being represented, and relationships among them [121]. Thus, ontology has also been defined as a representational vocabulary [61,103]. A formal specification is machine-interpretability in a way that a machine can understand and process the specified conceptualization. This would enable ontology to be used for several purposes such as to make meaningful communication among their agents/users, facilitate interoperability, knowledge sharing & reuse among subsystems and, to distinguish the domain knowledge from operational knowledge [103]. These can be viewed as capabilities of ontologies [103,138] and consequently, ontologies are increasingly incorporated into information systems.

However, the above capabilities would not be a reality unless a good quality ontology is produced at the end of the development. This is because one trivial internal quality issue would cause multiple quality issues in the end product which may result in a high cost of debugging and fixing the issues and in turn loss of consumers’ trust. For instance, it has been revealed that the ontology-based system in the medicine domain has caused a 55% loss of information due to one missing explicit definition in the ontology [56,57,60,89,151]. Thus, it is required to sufficiently evaluate both the quality of the content of the ontology during the development, as well as, the ontology as a whole after the development [56]. This has been viewed as two aspects of ontology evaluation in [101,105,154] namely intrinsic and extrinsic that have been further discussed in Section 3. Moreover, it has been revealed that the availability of a widely accepted methodology or model, or approach for the quality evaluation of ontologies would be beneficial for producing a quality ontology [105]. However, such a methodology, model, or approach has not yet been identified for ontology quality evaluation as the way it is in the other related fields such as system and software engineering. For instance, the availability of widely agreed quality standards in software engineering such as ISO/IEC 9126 [78], ISO/IEC 9241-11 [76], ISO/IEC 25012 [75], ISO/IEC 25010 [77], and IEEE software quality standards [74] provide immense support in developing good quality software.

Fig. 1.

The abstraction process of specifying quality requirements.

Since this article is on the ontology quality aspect, first we examine quality from the general perspective. Quality is referred to as the fitness of a product or service or information for the needs of the intended users in a specified context [31]. Basically, the quality of a system indicates how far the system caters to user needs with respect to a particular context. The context (i.e., the context of use) defines the actual conditions under which a product (i.e., software/Information Systems (IS)) is used by intended users in performing a task (see Fig. 1). The context not only considers the technical environment of the system but also the user type (i.e., attitude, literacy, experience), the physical and social environment, the available equipment and tasks to be performed [31,127].

Quality characteristics are a set of properties that can be used to assess whether the intended user needs of a system or product are met. In software and system engineering, there are a set of standards [76–78] namely; ISO/IEC 2501n, ISO/IEC 2501n and ISO/IEC 2503n [77], which describe how the quality characteristics to be evaluated are identified with respect to the user needs of a system. According to the standards, the quality requirements, which can also be viewed as a set of quality characteristics, should be elicited from user needs at the requirement specification phase [17,141] (see Fig. 1). To this end, a quality model plays a significant role that consists of a proper set of quality characteristics and associated measures. The quality model provides a framework for determining characteristics in relation to user needs [75,77,127]. The identified quality characteristics can be evaluated at the intrinsic and extrinsic levels of a product using their respective measures. This evaluation helps to ensure that the required quality is being achieved across product development. Figure 1 shows the abstraction process of deriving the quality requirements and how a quality model supports that process. The significance of a quality model is further elaborated in Section 3 with an example.

If we apply the process in Fig. 1 to the ontology context, it is possible to define the user as a person who interacts with an ontology-based IS. Context of use is the user type (as stated above), the environments (social, physical, technical) that the users may involve and the task to be performed through the ontology. User needs are the set of requirements that are expected to be fulfilled by the ontology-based IS. Additionally, the quality requirements are a set of overall quality requirements* for the ontology-based IS elicited from the user needs in a particular context of use. From these overall quality requirements, it is possible to derive the quality requirements related to the ontology in a system. Thereafter, the ontology quality requirements can be specified using quality characteristics with the support of an ontology quality model. This concept further discusses in detail in Section 3 (see Fig. 5).

It can be understood that quality models play a significant role in determining the quality characteristics of a system. When considering the ontological point of view, there are several quality models which have been proposed for ontology quality assessments. However, none of them is widely accepted in practice. This could be because some of the models such as the semiotic metric suit [24], the quality model of Gangemi et al. [48,49], and OQuaRE [40] are generic. Thus, it is required to additional efforts to customize and add absent characteristics which are significant to the specified context. Moreover, some of the quality models such as OntoQualitas [114] and the quality model of Zhu et al. [110] are specific to a domain (i.e., domain-dependent). Therefore, these models do not guarantee to work well for domains other than their specified domain. Consequently, the researchers and practitioners face the following difficulties.

Thus, there is in fact a requirement to perform a thorough analysis on the domain of ontology quality models and problems. As a result, a systematic review was performed to streamline the findings of the existing quality problems and thereby produce a conceptual quality model that is useful for researchers and practitioners to understand the characteristics, attributes, and measures to be considered in different aspects of ontology quality assessment. Moreover, in ontology development, ontology can be developed entirely from scratch or can be reused from the existing ontologies or a combination of both. However, ontology quality assessment is performed against the same set of requirements (i.e., characteristics) specified in a particular context regardless of whether the developed ontology is built from scratch or reused [102]. Thus, the conceptual model proposed in this study is helpful for assessing the quality of ontologies irrespective of how they are developed (i.e., from scratch and/or reused). Furthermore, when discussing the conceptual quality model, we focus on web ontologies which are built upon one of the standard web ontology languages such as RDF(S) [136] and different variants of OWL [135]. However, some quality characteristics such as compliance, consistency, completeness and accuracy can also be associated with other ontology languages. Moreover, we do not distinguish between TBox (i.e., ontology structure) and ABox (i.e., knowledge base) when discussing the quality characteristics. The term ontology is used to refer to both TBox and ABox.

The rest of the article is structured as follows; Section 2 presents the survey methods that we have followed. Section 3 provides preliminaries and conceptualization of ontology quality assessment to understand the concepts that have been described in the rest of the sections. The overview of the existing ontology quality models is presented in Section 4. In Section 5, the quality characteristics, associated quality measures, and definitions for each quality characteristic are described. Section 6 provides a discussion of the findings of the survey. Finally, Section 7 discusses the important gaps related to ontology quality models/approaches and Section 8 concludes the survey with highlights of future works.

2.Survey methodology

Firstly, a traditional theoretical review was performed by retrieving the related ontology quality surveys and the literature reviews, to explore the ontology quality evaluation and then, to identify the possible quality models, including significant ontology quality characteristics and measures, and the relations among the characteristics. Few attempts have been carried out to present such contributions [51,93,108,112,140]. However, we consider that they are not comprehensive surveys as they have not clearly defined: the research gaps and questions that they have addressed and the survey methodology that they have followed. Furthermore, these surveys have provided only a general overview. Moreover, the discussions were limited to a set of characteristics proposed in [48,57,58,62,142]. In addition to that, few comprehensive surveys on ontology evaluation have been identified (see Table 1) in which the main focus is on ontology evaluation in broader aspects, not specifically for ontology quality problems.

We conducted the systematic review by following the methodologies described in [23,83] with the objective of addressing the issues (i), (ii), and (iii) stated in Section 1. Thus, the review includes a summarization of ontology quality problems and models: recognition of characteristics and measures. Eventually, the aim is to present a conceptual model that provides a basis for researchers to retrieve the quality characteristics upon the quality requirements. To achieve these objectives, the following research questions were derived;

Then, the search terms were identified based on the research questions such as quality, ontology, ontology quality, assessment, evaluation, approach, criteria, measures, metrics, attributes, characteristics, methods, and methodology, and trial searches were executed using various combinations of the terms. Thereafter, the following combination was used to retrieve the relevant papers from the selected digital databases, journals, and search engines (see Fig. 2).

These search combinations were performed according to the instructions given in each digital database under the advanced search.

Fig. 2.

Main steps of the review process and number of articles retrieved.

The following inclusion and exclusion criteria were defined; to reduce the likelihood of publication bias, to select the recently discussed studies and to keep the sample size down to make the review easier to handle. Accordingly, the defined inclusion and exclusion criteria were used to select the candidate studies at the screening and eligibility phases (see Fig. 2),

During the screening phase, the candidate studies were retrieved by performing a keyword search on titles and abstracts. Then, the duplicate studies were removed using the reference management tool (i.e., Mendeley). Thereafter, the abstracts were reviewed to filter the relevant studies. There are some cases wherein the abstract is vague and difficult to understand the real contribution. In this case, the papers were forwarded to the second round for further review of the introduction and conclusion as suggested in [23]. Altogether, 128 papers were forwarded for the full paper review. During this stage, forward and backward citations were searched and included the appropriate studies to report. It has been identified that some papers have two versions: journal and conference entitled to the same authors. In such a situation, the journal paper was selected. Finally, thirty (30) papers mentioned in the Appendix were selected for the reporting phase and they have been considered as the core papers in the survey. In addition to that, the milestone papers, which were captured through the backward citation searching (i.e., before 2010), have also been taken into account for the discussions (see the Appendix).

3.Preliminaries and conceptualization

In the context of quality, a number of ad hoc definitions and inconsistent terminologies appear in the literature leading to terminology misapplications and misinterpretations [13]. Thereby, in this section, the concepts and the terms that are relevant to our discussion are briefed to avoid miscommunication and thus to maintain consistency. Additionally, the conceptualization of ontology quality assessment is provided based on the existing theories and Table 2 presents a list of terms used in this article.

Terminologies related to quality models Quality models provide the basis for specifying quality requirements and evaluating the quality of an entity (i.e., software, tools, ontology, part of the software) [75,77]. In software engineering, quality models are twofold, relational (i.e., non-hierarchical) and hierarchical [55]. The relational model presents the correlation among the quality characteristics and this type of model has not been proposed for the ontology so far in the existing works. Only, the hierarchical models can be found which have usually four levels (see Fig. 3). The top-level (i.e., the first level) consists of a set of characteristics that are further decomposed into sub-characteristics at the next level (i.e., the second level). The third level would consist of associated attributes of characteristics/sub-characteristics. These attributes have measures, which lay at the bottom level (i.e., the fourth level), that can be used to assess the entity either quantitively or qualitatively. Thus;

Characteristics (i.e., Criteria): describe a set of attributes. An attribute is a measurable physical or abstract property of an entity [84]. In the ontology quality context, an entity can be a set of concepts, properties, or an ontology [80].

Measure (i.e., metrics): describes an attribute formally and assesses them either quantitively or qualitatively [11].

Fig. 3.

Association between characteristics, attributes, and measures.

There are no definite requirements to have sub-characteristics for each characteristic or else characteristics/sub-characteristics should have attributes. Thereby, a characteristic can directly be related to measures without a set of attributes (see Fig. 3). Moreover, the same attributes/measures can be used to assess many characteristics. This has been further illustrated in Fig. 4 with the selected ontology characteristics. For instance, complexity, which is one of the ontology characteristics, describes the properties of the ontology structure (i.e., taxonomy, non-taxonomy) [48,49,116,142]. It is associated with several attributes such as size (A1), depth (A2), breadth (A3), fan-outness (A4) and density (A5) of an ontology [50] and each attribute has a set of measures. If the size attribute is considered, the related measures are the number of classes (M1), the number of relations (M2), and the number of instances (M3). Thus, the associated levels of complexity related to quality models are <characteristics, attributes, measures>.

Fig. 4.

An example for a simple quality model.

Moreover, the measures of the attributes: size, depth, breadth, and fan-outness have also been used to assess the cohesion (A6) attribute of the modularity characteristic [132,153].

If we consider the consistency characteristic of an ontology, it can be further derived into two sub-characteristics namely, internal consistency (SC1) and external consistency (SC2) [58,153]. Internal consistency contains direct measures such as circularity errors (M4), the number of multi-parent classes (M5), and the number of misused disjointness (M6) [56,58]. Thus, with respect to that scenario, the associated levels in the quality model are <characteristics, sub-characteristics, measures>. However, external consistency can be measured through attributes interpretability (A7), clarity (A8), and precision (A9) [96,131]. Each attribute has a set of measures that have been presented in Table 8. Thus, it covers levels <characteristics, sub-characteristics, attributes, measures>.

Moreover, sub-characteristics also can be further derived into another set of sub-characteristics also called primitive characteristics [19] as the way it is in OQuaRE [40]. Additionally, it can be observed that some of the quality models consist of another higher level called dimensions or factors on top of the characteristics-level. It has been proposed to classify the characteristics/attributes which are related to similar aspects of the ontology. For instance, the quality model of Gangemi et al. [48,49] contains three dimensions namely; structural, functional and usability-related. Similarly, the quality model of Zhu et al. [153] has classified the characteristics under the content, presentation, and usage dimensions. However, there is no clear definition is given for the dimension in literature, thus, the term: dimension has been used interchangeably to define the term characteristics. For instance, in [146,152], the term: dimension is used to define characteristics. However, in our study, we use the terms in the order: dimensions, characteristics, sub-characteristics, attributes, and measures to describe the structure of a quality model whenever it is necessary.

Ontology layers (i.e., levels) The researchers have discussed the ontology quality layer-wise, or level-wise by concerning an ontology as a multi-layered vocabulary [22,57,108,112,142]. Initially, three layers to be focused on have been proposed in [57], namely: content, syntactic & lexicon, and architecture. Later, this was expanded by including the layers: hierarchical and context [22,142]. The syntactic layer considers the properties related to the language that is used to represent the ontology formally [22,57]. The hierarchy or taxonomy layer focuses on the properties related to the taxonomic structure (i.e., is-a relationship) of ontologies [22]. A number of measures have been defined related to the taxonomy layer [42,48,50,54,116]. These measures are often useful to observe the extent to which the concepts in an ontology are spread out in relation to the root concept of that ontology [42,116] and to track the evolution of ontologies easily [142]. The authors in [42,116] have shown that taxonomic measures such as maximum depth/breadth, average depth and depth/breadth variance are good predictors of ontology reliability. The architectural layer is known as the structural or design layer in [22,108]. It considers whether an ontology is modeled based on the pre-defined design principles and criteria [22,57]. The lexicon layer is also named as vocabulary or data layer [22,142], which takes into account the vocabulary that is used to describe concepts, properties, instances, and facts. The non-hierarchical relationships and semantic elements are considered under the semantic layer. To evaluate the vocabulary, architecture and semantic layers, some understanding of domain knowledge is required. The context layer concerns the application scope that the ontology is built for. Mainly, it determines whether the ontology satisfies the application requirements as a component of an information system or a part of a collection of ontologies [22]. We have performed a separate analysis on the ontology evaluation layers and their relationship between the evaluation approaches (i.e., methods): task-based, human-based, data-driven, and golden-based, which can be found in [147] for more detail.

Ontology development methodologies and design patterns Ontology methodologies and design patterns and principles provide guidelines for knowledge modeling and representation paving the path for accurate knowledge capturing and defining. This enables ontology developers to achieve the quality characteristics emphasized in this study such as compliance, consistency, completeness, and conciseness. Due to this reason, it is necessary to explore the development methodologies, design patterns and principles. However, discussing them in detail is out of the scope of this paper.

There are several methodologies introduced for ontology development such as Cyc [92], Uschold and King’s method [139], TOVE [64], METHONTOLOGY [59], SENSUS [128], On-To-Knowledge [120] and DILIGENT [110]. Among them, only Uschold and King’s method, TOVE and On-To-Knowledge have discussed ontology evaluation as a phase to be performed after the development of the entire ontology. The METHONTOLOGY mechanism performs the evaluation at every phase of ontology development. However, none of these methodologies have neither described nor revealed the ontology evaluation methods in detail [27,69]. More details of the comparison of these methodologies can be found in [27,29,87]. In addition to that, agile ontology development methodologies have been introduced by adopting agile principles and practices in software engineering. These include methodologies such as SAMOD [109], XD [18,30], AMOD [1] and CD-OAM framework [129]. Out of these, only SAMOD and XD discussed the evaluation (i.e., model test, data test, unit test) of ontologies at least in a nutshell. The significance of XD (i.e., eXtreme Design with Content Ontology Design Patterns) is the use of ontology content design patterns to construct ontologies that instinctively support to prevent of making common modeling mistakes. This, in turn, directs ontologists to construct a quality ontology, particularly assisting inexperienced ontologists [18].

Ontology Design Patterns (ODPs) are reusable modeling solutions to recurrent ontology design problems [47]. Several groups have proposed a set of design patterns. For instance, the authors in [6] have proposed seventeen ODPs in relation to the biological knowledge domain. They have classified ODPs into three groups namely (i) extension ODPs which provide solutions to bypass the limitations of OWL such as n-ary relations, (ii) good practice ODPs which support to construct of robust and cleaner design (iii) domain modeling ODPs which provide solutions for concrete modeling problems in biology [6]. Moreover, the semantic web best practices and deployment working group has introduced ODPs which include n-ary relations, classes as property values, value partitions/sets, and simple part-whole relations. In addition to that, the group ontologydesignpatterns.org suggests more than two hundred patterns and principles which have been categorized into six families namely; content ODPs, structural ODPs, correspondence ODPs, reasoning ODPs, presentation ODPs and lexico-syntactic ODPs [9]. The reuse of the patterns and principles can be determined at the ontology design phase with respect to the requirements of the ontology development and it makes ontology modeling faster [71,102]. To this end, the authors in [18,71] have shown through experiments that pattern-based ontology development avoids making common modeling mistakes. This in turn improves the quality and usability of ontologies.

Furthermore, OntoClean has proposed a set of principles (i.e., rules) that helps to identify problematic modeling choices associated with taxonomy relationships [65]. Mainly, the rules of OntoClean have been constructed upon a set of metaproperties related to taxonomy namely; identity, unity, dependence, essence and rigidity. By embedding these principles, an ontology-driven conceptual modeling language has been proposed namely OntoUML [16,71,106]. Moreover, OntoUML provides a set of design patterns that accelerate ontology modeling. In addition to that, it describes a set of anti-patterns to be avoided in modeling ontologies.

Fig. 5.

Conceptualization of ontology quality evaluation [148].

Ontology evaluation space An ontology-based information system usually consists of many components. The quality of each component of the system would influence the overall quality and in turn usability of the system which is often critical for business success [17,35]. As explained in Section 1, the quality requirements which are to be achieved through a system or each system component should be traced from user needs also known as business requirements (see Fig. 1). Specification of quality requirements can be viewed as a connected flow starting from user needs to the internal quality of a system. For instance, quality requirements can be elicited from user needs and they can also be defined as a set of quality requirements to be achieved by a system when it is in use. This is also defined as quality in use (see Fig. 5) [17,75]. Then, this set of quality requirements can be used to determine the external quality requirements of each component of a system, accordingly, determines the internal quality requirements. These quality requirements (i.e., Internal/External) can also be expressed as a set of characteristics with associated measures. For that, a quality model can be used that provides a framework for determining characteristics/sub-characteristics in relation to the quality requirements. Then, these characteristics/sub-characteristics can be evaluated using the relevant measures to ensure whether the desired quality in each stage is achieved (see Fig. 5).

For an ontology-based system, ontology quality requirements also need to be distinctly identified from the overall quality requirements of the system [141]. Based on that, the external and internal quality requirements to be achieved from the ontology can be derived. The external and internal quality have been discussed as extrinsic and intrinsic evaluation aspects of ontologies in [101,105]. Figure 5 presents the flow of specifying the ontology quality requirements with the associated evaluation aspects when an ontology is a component of a system [148]. It should be noted that the quality requirements related to the other software and hardware components of the ontology-based system have not been presented in Fig. 5.

The extrinsic evaluation considers quality under two aspects namely domain extrinsic and application extrinsic. The ontology quality requirements elicited from the user needs can be separated under these aspects [125,141]. For instance, quality requirements associated with domain knowledge are considered under the domain extrinsic aspect. On the other hand, ontology quality requirements that are specifically needed by the system that the ontology is deployed are considered under the application extrinsic aspect [101,105]. Moreover, the application extrinsic requirements are independent of the domain knowledge. In performing a quality evaluation under these aspects, an ontology is taken into account as a component of a system. Traditionally this evaluation is called “black-box” or “Task-based” testing [70,101] as the quality assessment being performed without peering into the ontology structure and content. Furthermore, this extrinsic evaluation has been defined as ontology validation in [121]. Accordingly, ontology is assessed to confirm whether the right ontology was built [121,142], mainly this is performed with the involvement of users and domain experts (see Fig. 5). Moreover, this has also been considered as the context-layer evaluation in [22,142].

The intrinsic evaluation focuses on the ontology content and structural quality to be achieved in relation to the extrinsic quality requirements. To this point, ontology is considered as an isolated component separated from the system. At this stage, ontology quality can be evaluated under two aspects: domain intrinsic and structural intrinsic [101], mainly performed by ontology developers. In the structural intrinsic aspect, the focus is given to the syntactic and structural requirements (i.e., syntactic, structural, architectural layers) which involve the specified conceptualization such as language compliance, conceptual complexity, and logical consistency. In the domain intrinsic aspect, ontology quality is evaluated with reference to the domain knowledge that is required for the intended needs of users [101]. Thus, ontology developers need to get the assistance of domain experts to evaluate this aspect. Moreover, the semantic, vocabulary and architectural layers of ontologies are evaluated under this aspect. Furthermore, the quality evaluation performed under the structural and domain intrinsic aspects are similar to the paradigm of “white-box” testing in software engineering [70]. Thus, from the intrinsic aspects, verification is being done to ensure whether the ontology is built in the right way [121,142].

The quality requirements identified under each aspect of the ontology evaluation space can be associated with the corresponding set of characteristics together with measures. Then, those characteristics can be measured in order to ensure whether the defined quality requirements are being achieved through ontology development. In the following section, we illustrate how the quality characteristics can be derived in relation to an ontology requirement using a use case in agriculture.

Use case We considered an ontology-based decision support system that provides pest and disease management knowledge for farmers. For simplicity, we assumed that ontology quality requirements have already been elicited from user needs and as such consider that the requirement specified below is the key quality requirement of the system.

From the extrinsic aspect of the ontology, the mentioned quality requirement is associated with domain knowledge in agriculture. Thus, it is a domain extrinsic quality requirement and it discusses the accuracy of knowledge to be provided through the ontology. At this stage, the accuracy of the knowledge can be observed in terms of the accuracy of answers given for the competency questions [150]. Moreover, accuracy should be evaluated with the assistance of domain experts and users. With respect to the identified extrinsic quality requirement, it is required to define what set of characteristics should be met from the intrinsic aspect. For that, we need to utilize techniques such as ROMEO [150] and GQM [11] that support deriving the intrinsic quality requirements from the higher-level (i.e., extrinsic) quality requirements. By adopting the ROMEO methodology [150], the following questions are formulated to derive the intrinsic quality requirements. For instance, to provide accurate pest and disease knowledge for farmers, domain knowledge should be correctly modeled for an ontology. For that, all required axioms11 with regard to pest and disease management should be correctly defined in the ontology. Accordingly, the question: Q1 can be derived. In addition to that, the ontology should be free from internal contradiction to reason and produce the correct knowledge. Thus, the question: Q2 also can be derived with respect to the defined extrinsic ontology requirements. It is noteworthy that we have not described all the steps followed under the ROMEO methodology and only the derived questions are highlighted. However, an interested reader may refer to the article [148,150].

Based on the derived questions in relation to the extrinsic quality requirement, it is required to identify the associated characteristics and the relevant measures to be evaluated. In this situation, ontology quality models play a key role that supports identifying the corresponding characteristic/sub-characteristic in relation to the derived questions. However, there is no such a well-formed quality model for ontologies as highlighted in Section 1. Thus, to this end, we determined the characteristics based on the literature survey. For instance, Q1 discusses external consistency as it considers whether the defined axioms are consistent with the domain knowledge. Q2 discusses internal consistency as it considers whether there are any internal contradictions within the ontology definitions. After identifying the characteristics, the relevant measures to be evaluated through ontology development should be identified. This process would be easier if a well-formed quality model is available. For the use case that we have considered, external consistency can be evaluated using the measure of precision, as explained in [150], which provides a ratio between the correctly defined axioms (O_i) and the total axioms defined in the ontology (O_a). The correctness/consistency of axioms (O_i) can be determined against a frame of reference (F_a) such as a valid corpus or the standard ontology [150]. Precision values can range from zero (0) to one (1). If the value of precision is zero (0) then it implies that the particular ontology has not correctly captured any of the axioms defined in the frame of reference. If the value of precision is 0.6 then it implies that the ontology has correctly captured 60% of the axioms with respect to the frame of reference.

If the value of precision is one (01) then it implies that the particular ontology has correctly captured all axioms with respect to the considered frame of reference. Meantime, it can be stated that the particular ontology has correctly represented the relevant knowledge of the domain.

Moreover, internal consistency can be assessed by observing the number of logical inconsistencies with the support of a reasoner. In addition to that, the measures such as the number of circularity errors, the number of subclass partitions with common classes and the number of partitions with common instances can be observed (see Table 7). These measures are considered as a set of pitfalls that could lead to wrong inference [58,111,114] and they can be detected using the tool OOPS! [111].

In this way, for the rest of the ontology quality requirements, the relevant ontology characteristics can be determined with the support of a quality model. This use case is further explained in the article [148] to illustrate how the other ontology characteristics (i.e., coverage, comprehensibility) and their measures are derived for the ontology requirements. These characteristics can then be measured through ontology development to ensure that the ontology being modeled is of good quality.

4.Ontology quality models

5.Classification of ontology quality characteristics

Several characteristics and measures have been discussed in the selected papers. From that, fourteen (14) significant characteristics: compliance, complexity, internal consistency, modularity, conciseness, coverage, external consistency, comprehensibility (i.e., intrinsic point of view), accuracy, relevancy, functional completeness, understandability (i.e., extrinsic point of view), adaptability, efficiency, were identified. Then, the identified characteristics were grouped under the four evaluation aspects which also can be defined as dimensions namely: Structural intrinsic, Domain intrinsic, Domain extrinsic and Application extrinsic. These dimensions were derived based on the ontology evaluation space as described in Section 3. Moreover, this model has the same nature as the ISO/IEC 25012- Data Quality model [75] which consists of only two categories namely; inherent data quality, and system-dependent data quality. However, we identified and defined three main categories for ontology quality viz: inherent ontology quality, domain-dependent ontology quality, and application-dependent ontology quality (see Table 6). The structural intrinsic aspect attaches to the inherent ontology quality. The domain intrinsic and domain extrinsic aspects are grouped under the domain-dependent ontology quality. Finally, the application extrinsic aspect was mapped with the application-dependent ontology quality.

Table 6

Ontology quality model (the gray-colored characteristics were adopted from the data quality standard: ISO/IEC 25012)

In addition to the identified characteristics through the survey, a set of characteristics that can be applied to ontology were adopted from ISO/IEC 25012 namely currentness, credibility, accessibility, availability and recoverability [75]. Moreover, other two time-related characteristics: timeliness and volatility were identified with the characteristic: currentness, in which, timeliness depends on both currentness and volatility characteristics [152]. Altogether, twenty-one (21) characteristics were mapped with the ontology evaluation space. Each of them is explained in the following sections and definitions were derived from the discussed theories and the standard ISO/IEC 25012. Moreover, the associated measures for each characteristic have been presented in tables in the respective sections.

6.Discussion

In our survey, we selected 30 core papers for review and these papers consisted of sixteen journals, eleven conference papers, and three chapters. The distribution of articles across the years is presented in Fig. 6.

Fig. 6.

Distribution of the core survey papers across the years.

In Table 12, the number of characteristics considered in each article was mapped along with the characteristics defined under the evaluation aspects in Table 6. The authors in [102] have not specifically discussed a set of characteristics. However, they have provided five high-level characteristics covering a set of questions attached to the ontology quality assessment. For instance, they [102] discuss high-level characteristics namely intelligibility (i.e., can humans understand the ontology correctly?), fidelity (i.e., does the ontology accurately represent its domain?), craftsmanship (i.e., is the ontology well-built and are design decisions followed consistently?), fitness (i.e., does the representation of the domain fit the requirements for its intended use?), and deployability (i.e., does the deployed ontology meet the requirements of the information system of which it is part?). Under each characteristic, a concise description of the quality to be focused on has been provided. The article [8] has evaluated ontology named U ontology (i.e., representation of the ontology usage analysis domain) by adopting eight characteristics proposed in [142]. Nevertheless, it is unclear how the considered characteristics (i.e., adaptability, conciseness, and fitness) were evaluated.

In summary, the least discussed characteristics in the selected studies are credibility, volatility, efficiency, accessibility, availability, and recoverability (see Table 12). None of the selected studies has considered timeliness and currentness for ontology quality assessments. However, there were frequent discussions on the characteristics compared to others in descending order: complexity, coverage, external consistency, internal consistency, modularity, compliance, and comprehensibility (see Table 12). Most of them come under the structural intrinsic aspect. Accordingly, it can be observed that the existing studies have provided less attention to the extrinsic aspects of the ontology, particularly after the ontology is deployed in a system. This issue has further been realized when analyzing the evaluation tools and methods (see Section 7.4). From that analysis, it can be shown that there are many tools available for evaluating intrinsic ontology quality, but only a few tools or methods are available for evaluating extrinsic quality.

When considering the terms and definitions of ontology quality, none of the works has made a clear distinction between the concepts of quality characteristics, attributes and measures from an ontology quality point of view. Because of this issue, ontologists have used these terms interchangeably. For instance, although clarity has been defined as an evaluation characteristic/criterion in [62], in [24,114], it has been defined as an attribute. Additionally, it can be seen that various definitions for one quality characteristic have been provided in the literature (see Section 5). This can create confusion among ontology developers and researchers, as they may not know which definition to use or which one is the most appropriate for their situation.

Another significant issue in the existing studies is that, except for the research [114,153], all other studies have used a common set of characteristics without taking the context of use and the user needs into account when defining ontology quality characteristics. As highlighted in Section 1, quality is the fitness of a product for the intended use (i.e., requirements/needs) [31]. From the ontological point of view, ontology quality is the fitness of the representation of the domain knowledge for its intended use [102,114]. Therefore, the quality characteristics which are to be achieved by ontologies should be specified with respect to the intended needs in the specified context of use. Based on this fact, it can be stated that all the characteristics presented in Table 6 are not equally important for all contexts of use. However, the characteristics such as compliance, internal consistency, external consistency, coverage, accuracy, functional completeness and timeliness can be viewed as core characteristics that are required for ontologies regardless of the context of use. In addition to these, there are a set of characteristics that are necessary to achieve through an ontology with respect to the intended needs and those characteristics should be properly identified. For instance, when considering the ontology-based decision support system in agriculture, relevancy is a crucial factor as stated in [145]. Moreover, the authors in [81] have exemplified that the modularity characteristic is not applicable to all ontologies and it depends on the needs of users and applications. In addition to that, authors in [114,150] have stated that conciseness of ontologies is not a required characteristic when an ontology is modeled for extensive coverage of the domain. Accordingly, it can be understood that the necessary and sufficient quality characteristics for an ontology should be identified in relation to the needs of the considered use case (i.e., the context of use) [114,150]. Then, these identified characteristics should be assessed across the ontology development life cycle [102].

7.Important gaps identified through the survey

In light of our survey, the important research gaps further to be investigated have been identified such as (i) the absence of relational quality models for ontologies, (ii) the lack of systematic approaches for ontology quality specification and evaluation (iii) the lack of detailed guidelines for selecting the correct modeling choices at the conceptual level (iv) the lack of methods and tools for assessing the extrinsic quality. The subsequent sections further explain these important gaps with the possible solutions.

8.Conclusion and future works

The findings of a systematic review of ontology quality assessments were reported in this article. The review considered thirty (30) papers published from 2010 to 2020 and six (06) milestone papers published before 2010 including four (04) other significant papers. As a result, around forty (40) papers were reviewed in total (see the Appendix). The main contribution of this article is that we proposed a quality model for ontology quality assessment (see Table 6) with formalized definitions of the characteristics and related measures. Nineteen characteristics were discovered primarily in relation to the four aspects of the ontology evaluation space. Out of these nineteen characteristics, timeliness was linked to two additional characteristics: currentness and volatility. Thus, altogether twenty-one definitions were derived. The proposed quality model (see Table 6) would provide an underpinning for ontology quality assessment and further experiments are required for a more complete model with a balanced set of characteristics, thereby it can be adapted for any domain with minimum amendments. Additionally, it is vital to empirically explore the effect (i.e., positive, negative) of changes in one characteristic to another that has not been so far discussed. Instead of that, currently, researchers have made assumptions about the correlation between the characteristics that cannot be acknowledged without rigorous experiments as they can be varied in the context where the ontology is built for. Based on the comparison of the previous works, it can be observed that none of the quality models and approaches covered all characteristics in the ontology aspects, nevertheless, a wide range of characteristics have been discussed in OQuaRE [40]. However, it does not support evaluating the semantic features of an ontology and the proposed attributes are subjective. Moreover, there is limited evidence related to the quality evaluation in the extrinsic aspect of ontologies, thus, more research on the extrinsic edge is required. In the next step, the aim is to empirically evaluate a set of ontologies that are modeled for a specified task with the identified characteristics and afterward, to propose a systematic approach for ontology quality specification and evaluation concerning a use-case in a selected domain.