MuHeQA: Zero-shot question answering over multiple and heterogeneous knowledge bases

3.1.1.Keyword discovery

Our goal in this step is to identify not only named entities but also concepts which will later allow extracting related information to the question from each of the KGs. Unlike other approaches that need to correctly fill the query template with the data extracted from the question, our goal is based on a textual summary that should be as rich as possible to obtain the answer with guarantees. In particular, the focus is on broadening the entity definition where possible (e.g. ‘ABC1 protein’ instead of just ‘ABC1’ from the question ‘what does ABC1 protein transport?’), and identifying the key concept when no entity is found (e.g. ‘overnight delivery’ from the question ‘in which country was overnight delivery filmed in?’). Our method is based on the PoS tags to discover keywords in a query. It uses the ‘flair/pos-english’ part-of-speech tagging model for English that ships with Flair [3] at Hugging Face,44 and organizes the PoS categories in levels of relevance to guide the keyword composition. At the first level are the categories that most commonly define an entity, such as Noun (NN), Proper Noun (NNP) and Cardinal Number (CD). At the second level are the categories that may characterise the entity, such as Adjective (JJ) and Coordinating Conjunction (CC). And the last level establishes the categories that invalidate the keyword to represent a single entity, such as preposition or subordinating conjunction (IN), Wh-determiner (WDT), verb (VBx) or particle (RP). PoS labels are assigned sequentially as a state machine (see Fig. 2) and keywords emerge when terms from the first level are together or next to some from the second level, but not from the third level. In the previous question, ‘in which country was overnight delivery filmed in?’, the keywords ‘country’ and ‘overnight delivery’ are identified from the labelling: in (IN) which (WDT) country (NN) was (VBD) overnight (JJ) delivery (NN) filmed (VBN) in (RP)?.

Fig. 2.

PoS categories defining a keyword in a question.

3.1.2.Resource identification

Once the keywords are identified, they have to be linked to available resources of the Knowledge Graphs to obtain related information that may help answer the question. As before, existing approaches are mainly focused on accurately identifying the resources to fit the query template. In our case, resource candidates should be prioritized not only for their textual similarity to the keyword, but also for their relationship to the question. The creation of vector spaces where each resource is represented by its labels [14] is discarded since one of our assumptions is to avoid the creation of supervised models that perform specific classification tasks over the KG (i.e. prior training). Our proposal does not require training datasets since it performs textual searches based on the terms identified in the query using an inverse index of the labels associated with the resources. At this point it should be noted that the more descriptive the property labels in the KG are, the better the text-based search will perform. A rank strategy for the resource candidates based on three types of textual similarity is defined: (1) at the name level; (2) at the property level; and (3) at the description level. Equation (1) states the relevance of a KG resource (res) based on the semantic similarity (sim(x,y)) between the given question (q) and the resource name (resname), the resource description (resdesc) and the resource property labels (respropi). The list of resource candidates is ordered from highest to lowest relevance. Different strategies to select the most relevant ones can be considered.

As already pointed out, unlike most existing approaches, the NLQ is not translated into a formal query (e.g. SPARQL) to retrieve the answer. Instead, the available properties in the KGs are queried for the most relevant resources mentioned in the query using the formal query language accepted by the underlying source (SPARQL, Cypher, …). This step is the only one anchored to the knowledge graph, and we minimize its dependency with the data schema since we only need to explore the values and labels of its properties, instead of traversing the relationships. KGs are organized by resource properties that can be RDF triples, relational table columns or facets. For RDF-based knowledge graphs, i.e. DBpedia or Wikidata, a unique SPARQL query (Fig. 3) is used to retrieve all the related information. These queries along with the rest of source code of the algorithm are publicly available for reuse.

Fig. 3.

SPARQL queries used to extract the properties of a KG resource.

The semantic similarity between the resource (i.e. name, property labels and description) and the query is based on the cosine similarity of their vector representations created with a sequence-to-sequence language model. Specifically, it is a sentence-transformers model55 created from the distilled version of the RoBERTa-base model and fine-tuned on a 1B sentence pairs dataset that maps sentences to a 768 dimensional dense vector space. This metric was considered because in the next step the evidence will be extracted using techniques based on the same representation model. An example about how it works on Wikidata is shown below. The most relevant resource candidates at the name level related to the keyword ‘Carlos Gomez’ identified in the question ‘What position does Carlos Gomez play?’ are: the researchers ‘Carlos Gomez’ (Q89898891), ‘Carlos Gomez’ (Q51944192), ‘Carlos A Gomez’ (Q91676432) and ‘Carlos M Gomez’ (Q40124092); the american actor ‘Carlos Gomez’ (Q949506); the mexican footballer ‘Carlos Gomez’ (Q203210); the Dominican Republic baseball player ‘Carlos Gomez’ (Q2747238); and the Chilean association of football players born in 1992 ‘Carlos Gomez’ (Q5750557). At the property level, i.e. comparing the cosine similarity of the resource property labels with the question, the list of most relevant candidates is reduced to the footballer ‘Carlos Gomez’ (Q203210) and the baseball player ‘Carlos Gomez’ (Q2747238). Finally, at the description level, i.e. the cosine similarity of the resource description with the question is added, the list is reduced to the footballer ‘Carlos Gomez’ (Q203210). This selective strategy based only on the best relevant scorers can be varied by manually setting the relevant top n (see Section 4 for more details).

In case of unstructured data sources, the resource identification step corresponds to the search for sentences or paragraphs where the keyword identified in the previous step is mentioned. More elaborate strategies could be considered, for example by selecting texts that contain terms related to the keyword, either syntactically or semantically.

3.1.3.Property verbalization

At this point the property-based comparisons from the previous step are reused to create a textual summary using only the most relevant properties (i.e. labels closest to the question). For each property, its value is obtained from the KG and expressed in a sentence. Although there are methods that are able to verbalise a relation between a value and an entity through a property, e.g. TekGen [2], the triple-to-text [39] method or UniK-QA [22], the source code is either not available or not currently operational for our approach so a basic solution has been developed for verbalising triples based on applying the following rule: ‘The <property> of <entity> is <value>’ (e.g. The duration of schizophrenia is chronic). Natural language phrases are then created from each property-value available in the knowledge graph that is related to the entity. Labels described in natural language associated with each resource are used: wikibase:label in Wikidata and rdfs:label in DBpedia. All these phrases make up the summary.

For example, an excerpt from the textual summary created from Wikidata for the question ‘What position does Carlos Gomez play?’ is as follows: “The position played on team / speciality of Carlos Gómez is defender. The given name of Carlos Gómez is Carlos. The member of sports team of Carlos Gómez is Atlético Potosino. The member of sports team of Carlos Gómez is Club León. The member of sports team of Carlos Gómez is Puebla F.C. The family name of Carlos Gómez is Gómez. The occupation of Carlos Gómez is association football player”. The order of appearance of the properties in the summary shows their closeness to the question. The former are more closely related than the latter. (i.e. ‘position played on team / speciality’, ‘given name’, etc.).

3.2.1.Document generation

The verbalisations of the properties in a single text document were combined. If additional unstructured data sources (e.g. texts) are available, this step involves the addition of the sentences or paragraphs where the keywords are mentioned in those data sources. It should be noted that language models limit the amount of text that they can process in each operation, so the document is splitted into smaller parts (no more than 512 words) in order to process them considering sentences as the minimum unit that cannot be broken.

3.2.2.Response retrieval

A text fragment (i.e. an evidence) from each textual part is extracted using EQA techniques, with the corresponding confidence. The EQA task consists in completing a text sequence (i.e. context) using a language model. The sequence is created by joining the textual content and the question. The model then infers the most likely text that would continue the sequence. This new text is the evidence to discover the answer to the question. Recent methods have even supported multilingual inferences [38], but the use of English language models based on bidirectional encoder representations from transformers (BERT) [8] was preferred to better understand its behavior with the text that is automatically generated from knowledge bases. Specifically, a general-domain language model66 trained on the SQUAD2.0 dataset [25] is combined with a more specific clinical-domain language model77 trained on a Covid-19 question answering dataset [27]. The extractions from the models was chained and the evidence was ordered according to its confidence score.

3.2.3.Candidate ranking

From all the evidences found in the previous step, the one that offers the highest confidence was chosen, although in other tasks a ranked list of potential answers is may also offered. For example, from the textual summary previously created for the question ‘What position does Carlos Gomez play?’, the most relevant evidence is ‘defender’ supported by the sentence: ‘The position played on team / speciality of Carlos Gómez is defender’, with the highest confidence score, equal to 0.99.

3.3.1.Query analysis

The type of the expected answer based on the user question is required. Most of the existing solutions consider domain-specific types, because they need to filter the resources they return in the response. However our approach works also with natural language in the response, so it is sufficient to distinguish the high-level answer categories: literal, numerical and boolean. Boolean questions (also referred to as Confirmation questions) only admit ‘yes’ or ‘no’ as answer (e.g. “Is chloroquine authorized by the European Medicines Agency?”). Numerical questions have a number as an answer (e.g. “How many types of intracavitary chemotherapy exist?”). And literal questions have a textual value as answer, which can be a string or date (e.g. “What side effects does abacavir commonly report?”). Answer type prediction was approached as a classification problem where each question is a sequence of words, so an existing fine-tuned BERT model trained with general-domain questions for category prediction [21] was used.

3.3.2.Answer generation

In the case that the type of the response is numeric or boolean, a post-processing of the response is required. For quantities a special character (usually comma) is assumed as separator, and the number of elements in the candidate response is counted. Note that given our approach a listing of the responses is may also provided. In the boolean case it is considered true when the confidence is above a threshold. In other case the answer will be created directly by joining the information about evidence, confidence and type of the answer.

This approach to provide not only the answer but also the text from which it is taken addresses our third research question (“How to provide evidence that supports the response that has been generated by a KGQA system?”), since it creates an evidence for the response in the form of a sentence that contextualise it together with a numerical value expressing confidence.

4.3.1.Keywords in a question

As described in Section 3.1.1, our method identifies the entities mentioned in a question along with the relevant terms discovered using PoS annotations. The method used is a standard part-of-speech tagging model for English that ships with Flair [3]. The performance of our method (i.e. MuHeQAkeyword) has been compared with state-of-the-art approaches based on the language models FLERT [29], BERT [8] and RoBERTA [40] for named entity recognition. Specifically, the implementations of the models that have been considered are those fine-tuned from the English version of the standard CoNLL-2003 Named Entity Recognition dataset, and that are available on the Hugging Face platform: ‘dslim/bert-base-NER’ based on BERT, ‘Jean-Baptiste/roberta-large-ner-english’ based on RoBERTA and ‘flair/ner-english-large’ based on FLERT. The evaluation consists of comparing the entities previously identified in questions retrieved from QA benchmarks with those identified by the method to be evaluated. Since in real QA environments the questions are not always well written (e.g. they may omit question marks, or do not use capital letters to name entities), the WikidataQA dataset [10] that has grammatically valid texts (e.g. Who is the author of One Piece?) has not only been used, but also the SimpleQuestions dataset that is more flexible and can omit punctuation marks, capital letters, etc. (e.g. what position does pee wee reese play in baseball).

Table 1

Performance identifying keywords in a question

Table 1 shows the results of the analysis. The highest precision is obviously achieved by the language models that have been fine-tuned to solve NER tasks, but at the cost of drastically reducing the coverage of entities. In KGQA systems this is a risk, since if no entity or key concept is identified in a question, it would remain unanswered. In addition, language models are also very sensitive to the grammatical characteristics of texts, as evidenced by their lower performance in the SimpleQuestions dataset where the texts do not use capital letters in the entities. Our approach based on the combination of entities with the key concepts inferred by PoS categories shows a better performance in both scenarios, where the recall is improved without severely penalizing the precision, thus improving the performance of the algorithm in absolute terms (i.e. f1-score).

4.3.2.Linking to KG resources

As described in Section 3.1.2, our method discovers relevant resources in a Knowledge Graph from the keywords identified in a question. It performs a textual search based on the terms in the keyword. The resources found in the KG are considered as candidates and, depending on the ranking criteria, will be more or less relevant to the question. In order to better measure the performance several configurations were proposed. MuHeQAall selects all candidates since it considers them to be equally relevant. MuHeQAbest selects only the best candidate or candidates (i.e. highest relevance score) after ranking them on the basis of equation (1). MuHeQAtopN ranks as before, but only selects the N best candidates.

The behavior of our linking method has also been evaluated on DBpedia and Wikidata. The textual searches were performed using the wbsearchentities action1313 in Wikidata, and the Lookup service1414 in DBpedia. They are services that use an inverse index to find a resource from the text of a label (e.g. name or description). In any other KG, an inverse index could be easily created from a resource label (e.g. name or description) to provide a similar service. This avoids any need for a training dataset, since any knowledge graph, whether large or small, general or domain-specific, can offer a search service based on labels. The only requirement is that there is at least one label for each resource and it is recommended that it is sufficiently verbose to find it.

Table 2

Performance when discovering Wikidata resources

The evaluation consists of comparing the resources found by our method based on a Natural Language question and the keywords previously identified. The SimpleQuestions dataset, previously processed to work in Wikidata and DBpedia, is used in this evaluation as it contains the keyword (i.e. entity name) and the KG resource for each question. The performance of our methods was compared with other existing solutions for linking resources in Wikidata (see Table 2) and DBpedia (see Table 3). Specifically, the spaCy Entity Linker1515 module was used to analyze performance in Wikidata, and DBpedia Spotlight1616 to make the analogous in DBpedia. Again, performance was measured in terms of (macro and micro) precision, recall and f-measure.

Table 3

Performance when discovering DBpedia resources

Regardless of the Knowledge Graph, it seems that the most promising approach is the one that considers only the first candidate (i.e. MuHeQATop1). It makes sense since the questions used in the evaluation are single fact questions and therefore only have one entity and one resource associated to them. The poor performance of the ranking-based approach on Wikidata (i.e. MuHeQAbest) is striking. It is not due to inability to discriminate the most relevant resources, because the recall is also low. We suspect that the reason is that the property labels in Wikidata are not as descriptive as in DBpedia, since we use them to measure their closeness to the question and help to select a candidate resource. And this is especially critical when the keyword, i.e. the label in the KG, is not specific enough. For example, given the keyword ‘Myocardial infarction’ identified in the question ‘Which Swiss conductor’s cause of death is myocardial infarction?’, our MuHeQAbest method linked it successfully to the Q12152 resource in Wikidata (i.e. myocardial infarction as an interruption of blood supply to a part of the heart) and to the ‘Myocardial_infarction’ and ‘Cardiovascular_disease’ resources in DBpedia. On the other hand, given the keyword ‘the medic’ identified in the question ‘what language is spoken in the medic’, the method wrongly considers the Wikidata resource ‘Q7750866’, which is ‘The Medical Journal of Australia’, instead of the resource ‘Q1517084’, which is the film ‘The Medic’. The reason is probably that the properties of the resource ‘Q7750866’ are more relevant to the question (in accordance with the relevance score described in Section 1) than those of the resource ‘Q1517084’. For example, the property ‘language of work or name’(P407) of the resource associated with the journal is more relevant to the question than the property ‘original language of film or TV show’(P364) of the resource associated with the film. However, the method correctly recognizes the resource ‘Medic_(TV_series)’ in DBpedia, which has the property ‘language’(dbp).

4.3.3.Answer composition

The performance of MuHeQA for generating answers to single fact questions by obtaining information from multiple Knowledge Graphs was measured. On this occasion the SimpleQuestions dataset was used on both knowledge graphs, Wikidata and DBpedia, to compare the performance with other state-of-the-art approaches such as STaG-QA [26], SYGMA [20] and Falcon 2.0 [28]. While Falcon 2.0 is not a KGQA system itself, it allows generating the SPARQL query based on the entities and relations it identifies [26]. Due to differences in the conceptual organization of the knowledge graphs behind the SimpleQuestions dataset, the directionality of equivalent predicates in Freebase (i.e. original) and DBpedia or Wikidata may differ. For example, the DBpedia predicate dbo:birthPlace takes a person as the subject and a location as the object, whereas the equivalent predicate in Freebase fb:location/location/people born here inverts the subject and object [4]. Table 4 shows the results considering only questions whose directionality holds between the Knowledge Graphs (i.e. ‘forward’ type questions).

Table 4

Performance based on knowledge graph-oriented QA

The results show that our approach offers a performance close to the best system, STaG-QA, and better than other approaches specific to KGQA. However, one of the weak points is the recall, which means that our approach has to improve in response elaboration. The answer is perhaps too straightforward, and we should be concerned with constructing more complex responses.

4.3.4.Unstructured data sources

Since MuHeQA also supports unstructured knowledge sources, its performance was evaluated on QA pairs based on text documents. The answers are composed from the set of passages that are considered relevant to a given question. In this scenario, while extractive QA-based approaches, such as ours, highlight the span of text that answers a query, generative QA-based approaches create answers based on pieces of text they consider relevant. Retrieval-Augmented Generation models (RAGs) are based on generative QA techniques and they have recently attracted the attention of researchers due to their high performance in QA tasks [18]. They accommodate fine-tuned language models in modular pipelines [13,16] or end-to-end architectures [30] to retrieve textual passages that are used to create the answers from a given question. Both approaches were compared to answer the questions provided on three domain-specific datasets. The COVID-19 dataset contains 1,765 QA pairs created from 5,000 full-text scientific articles selected from the CORD-19 corpus [34]. The News dataset contains 5,126 human annotated QA pairs based on 10,000 news articles selected from the NewsQA dataset [33]. And the Conversation dataset contains 3,260 QA pairs based on 10,000 conversations retrieved from the QAConv dataset [35]. Exact Match (EM) and F1 score were used as evaluation metrics. The EM score computes the word level exact match between the predicted answer and the real answer. The F1-score calculates the number of words in the predicted answer that are aligned with the real answer regardless of the order. The results are showed in Table 5.

Table 5

Performance based on document-oriented QA

The results show a similar behaviour to the evaluation based on knowledge graphs and, in general, offer high performance. The answers created by our algorithm are not as elaborate as those in the evaluation dataset, which were created manually, and this penalises the performance of our system. For example, given the question “How many children were infected by HIV-1 in 2008–2009, worldwide?”, the answer inferred by our system is “more than 400,000”, while the correct answer is “more than 400,000 children were infected worldwide, mostly through MTCT and 90% of them lived in sub-Saharan Africa”. This behaviour has already been shown previously when evaluating performance with graph-based knowledge bases and is one of the current limitations of our proposal. The creation of more elaborated responses is one of the future lines of work that we need to address. On the other hand, the accuracy is quite high, offering a promising overall performance. This makes us optimistic not only for the handling of documentary knowledge bases, but also for producing higher quality summaries from KGs that will allow us to achieve a performance like the one obtained for documentary datasets.