ImageSchemaNet: A framester graph for embodied commonsense knowledge

3.1.Defining image schemas

According to Johnson’s famous definition, “an image schema is a recurring, dynamic pattern of our perceptual interactions and motor programs that gives coherence and structure to our experience” [42]. For instance, playing with shape puzzles22 as an infant, represents early experiences of spatial boundedness and Containment. IS are directly meaningful experiential gestalts [43], that is, they are internally structured compositions of parts to form coherent, uniform wholes. These repeatedly experienced structures are considered to shape higher-level abstract cognition, such as language and commonsense reasoning. Although previous works, also providing forms of neural grounding [16], have proposed an IS inventory [11,58], a generally agreed upon final list is still open to debate and Johnson’s [42] caveat still stands:

To give an example, He just sails through life depicts life as a Container, through which we travel on our Path. By way of metaphorical projection, structures of physical source domains can be mapped onto abstract target domains, e.g. the inside (being alive), outside (not being alive), and boundary (birth, death) [46]. Johnson [42] and Lakoff [50,53] provided many linguistic and sensorimotor examples, as well as related high-level entailments without, however, formalizing their theory.

To provide a more formal account, Hedblom et al. [37] propose the unified metalanguage Distributed Ontology, Modeling and Specification Language (DOL) [62] to represent shared gestalt structures of seemingly unrelated image schemas as a family, that is, a set of interlinked theories. Such a gestalt grouping of experiential structures implies a distinction between primitive and complex types. To this end, the theory of Mandler and Pagán Cánovas [58], rooted in developmental psychology, was adopted to distinguish between spatial primitives, image schemas, and conceptual integrations. Spatial primitives are the very first preverbal building blocks that infants form to quickly compose more complex structures, i.e., spatial primitives are the parts that compose coherent unified wholes. These wholes (or spatial events) built from spatial primitives are image schemas. Finally, conceptual integration refers to the inclusion of non-spatial elements, such as emotions. Hedblom et al. [37] take up this initial definition and depict spatial primitives as roles participating in a frame’s image schema (e.g. the spatial primitives Source, Path and Goal are roles of the Source_Path_Goal image schema). Thereby, image schematic-structures, primitive or not, can be grouped based on experiential gestalt family resemblances. This initial formalisation is later extended as Image Schema Logic (ISLM) [36].

The natural language definitions provided in the works introduced in this section were taken as the basis to propose our ontological representation of image schemas. Instead of relying on first order logic or different calculi for this representation, we focus on a practical frame-based structure and a strong connection to lexical resources. The following presents a list of IS, for which ImageSchemaNet currently provides both a formalization and a lexical coverage, with natural language definitions from IS literature:

Examples of other frequently discussed IS are among others: Contact, Scale, Link, Balance, Object, Substance and Cycle. These IS are not yet in ImageSchemaNet because, compared to the above list, they are less documented in literature. This is due to various reasons. For example, Object is the abstraction of any sensorimotor experience of any bounded entity of the world, and this would result in considering all physical and, by metaphorical projection, all non-physical entities as activator of Object, which would not generate distinct, relevant knowledge. For other IS there is an ongoing debate on whether they should be considered IS in their own right, e.g. Contact is at times seen as a spatial primitive interacting with Support [57,58] or its right as being an IS itself is debated [36]. Thus, the list above represents the most agreed upon selection of IS from literature at the moment of writing.

3.2.Frame semantics and Framester

To connect our ontologies to linguistic examples of image schemas, we rely on formal representations of frames from FrameNet [64] and MetaNet [24] in Framester. Frames in a most general notion are (cognitive) representations of typical features of a situation. Fillmore’s frame semantics [18,19] has been most influential as a combination of linguistic descriptions and characterisation of related knowledge structures to describe cognitive phenomena [12]. Words or phrases are associated with frames based on the common scene they evoke. In FrameNet, frames are also explained as situation types and they have been used by Dodge to improve embodied construction grammar coverage [17]. In Framester semantics [22] observed/recalled/anticipated/imagined situations are consequently occurrences of frames.

Fillmore explicitly compares frames to other notions, such as experiential gestalt [52], stating that frames can refer to a unified framework of knowledge or a coherent schematization of experience. Thus, widely acknowledged frames provide a theoretically well founded and practically validated basis for commonsense knowledge patterns.

Framester [22,23] provides a formal semantics for frames in a curated linked data version of multiple linguistic resources (e.g. besides FrameNet, WordNet [60], VerbNet [45], BabelNet [63], etc.), factual knowledge bases (e.g. DBpedia [2], YAGO [78], etc.), and ontology schemas (e.g. DOLCE-Zero [26,27]), with formal links between them, resulting in a strongly connected RDF/OWL knowledge graph.

Framester can be used to jointly query (via a SPARQL endpoint33) all the resources aligned to its formal frame ontology.44 Framester has been used [24] to formalize the MetaNet resource of conceptual metaphors,55 based on FrameNet frames as metaphor sources and targets, as well as to uncover semantic puzzles emerging from a logical treatment of frame-based metaphors. Yet, an image-schematic analysis of MetaNet was lacking, and can be enabled by a refinement of the FrameNet imagistic foundation, which has only been sketched with respect to Framester’s prepositional knowledge [25] starting from [56].

4.1.ImageSchemaNet classes

:ImageSchema The :ImageSchema class represents the general concept of Image Schema, it is defined using the :bibRef property, quoting literature definitions, and it takes as instances image schemas whose activation is covered in the ImageSchemaNet ontology. Each IS is axiomatized as a gestalt structure, composed by at least 2 spatial primitives, and it is modeled as a kind of conceptual frame.

:SpatialPrimitive The :SpatialPrimitive class takes as instances the “first conceptual building blocks formed in infancy” as in [58], and represents them as semantic roles. The labels used respectively for IS and SP refers to well established and documented names used in literature, as for the Support IS, quoting their definition and provenance. When specific “official” names were not already given to entities, which existence was nonetheless implicitly or explicitly stated, we used labels extracted from empirical use case. For instance, in the aforementioned Support case, while literature is often mentioning examples involving its spatial primitives, no official name was available, and for this reason the Supporter and Supported SP were introduced from anew.

:IS_Profile The :IS_Profile class is used as in [38] and [65] to describe the collection of IS which are activated by some entity, sentence, situation or event. One of the relevant future developments stemming from our work is the automatic extraction of the image schema profile and the investigation of the conceptual nature of relations among IS in such a collection. The prominence of one particular IS in a set generated from, for example, a text string, refers to a form of frame compositionality as in [22], which could be determined by syntax as well as discourse structure, depending on term, sentence and text compositionality. The :IS_Profile class is particularly relevant here since it’s the class used for our evaluation system as described in Section 6.

4.2.ImageSchemaNet properties

All the activation declarations in ImageSchemaNet are realized via the :activates object property or its subproperties, which specify details about the way, layer, resource and type of activation. The meaning of :activates refers to some element that activates the cognitive substratum that is associated with an image schema. For instance, the verb to contain, the noun container, the frame Containment, and the frame element Container all activate the image schema Containment.

For this reasons, the following sub-properties were introduced in the graph:

In Section 7 we provide examples of cases from Section 6, in which the assertion of some synset as activator could be acceptable or debatable, and some others in which a specific type of activation has been crucial for detecting the correct IS. Some useful queries to explore the resource using the aforementioned properties can be found in Appendix B. In the following section we describe the SPARQL queries used to retrieve Framester entities activating an image schema or a spatial primitive.

5.1.Frame-driven activation

We have started looking for the frames activating an IS. The first search uses a non-disambiguated lexical unit (e.g. contain for the Containment IS) to retrieve all the senses and frames evoked by a lexical unit in isolation. For example, for contain, the searching process can collect all its senses, and their evoked frames. Based on sense inheritance hierarchies (as available in OWL versions of WordNet and other lexical resources), the search is extended to more specific or more generic senses of e.g. contain, so extending the set of evoked frames, and potentially activated IS. This kind of query is exemplified on Framester and can be found in Appendix A at the “Frames Activation Query” paragraph, in Appendix E, Fig. 6 as “Frame activation query” starting node, and in the OWL file as annotation of the :activates object property using the :operationalizedVia annotation property.

However, the amount of senses and related frames can be large, and we need contextual disambiguation in order to make it more precise. After performing the query, the selection of frames activating an IS is done manually, and after the iteration of the query for all synonyms and hyponyms, the first phase of frames activation search is closed, and we move to the frame element activation search.

5.2.Frame element-driven activation

Frame element activation concerns the activation of a spatial primitive, and can be performed similarly as with frames. This kind of query is exemplified by focusing on retrieving FrameNet frame elements of type “Core”, “Extra-Thematic” and “Peripheral”. After performing the query, the selection of frame elements is done manually, using as pivotal the set of frames selected in the step before, possibly enriching the set with further frames, not retrieved by the query in the first step. The query is available in Appendix A, in the “Frame Elements Activation Query” paragraph.

5.3.Lexical unit-driven activation

Activation from lexical material is a substantial part of the heuristic abstraction, and it is generated by automatically querying Framester knowledge base, asking for all the elements (typically WordNet synsets or VerbNet verb senses) that evoke a frame. The query is performed for all the frames retrieved and selected as activators by the Frame Activation query. The heuristic rule here is: if an entity evokes a frame, which activates an IS, than that entity should have some form of activation for the IS. The amount of elements retrieved may be considerable (for some IS, thousands of WordNet synsets). As a consequence, the synsets in the knowledge base are very useful for the coverage they provide in the populated ImageSchemaNet knowledge graph.

Of course, this coverage may contain some noise, since synsets are retrieved by making an inference from previous existing alignments in Framester, which may have different levels of confidence on their turn (there is a manually curated kernel of alignments, while other sets have used various heuristic rules). For example, both the “vase” and “absolutism” terms end up activating the Container image schema, because some synsets for theoretical concepts of philosophical doctrines or behavioural attitudes (e.g. “absolutism”) are aligned in Framester with a skos:closeMatch to the FrameNet frame Containing (because of a lower alignment confidence). In practice, “absolutism as a container” could be considered valid only when conceptual metaphors, e.g. IDEAS ARE OBJECTS, THINKING IS OBJECT MANIPULATION or CATEGORIES ARE BOUNDED REGIONS, are taken into account. A part of Section 7 discusses this and other critical examples. This query could be found in Appendix A at paragraph “Lexical Elements Activation Query”.

5.4.Semantic role-driven activation

Activation assertions to FrameNet frame elements are extended through the multiple sources of semantic roles present in Framester (VerbNet arguments, PropBank roles, WordNet tropes, etc.). Semantic roles in Framester are organized as a complex taxonomy with a small top level that helps integrating them, and getting to the activated IS. The activation of spatial primitives, modelled as semantic roles, is materialised via the :semantiRoleActivation property. Roles are retrieved with two queries, starting from top nodes of different graphs, in order to declare the activation of both general and specific roles. The queries are available in Appendix A in the “General Semantic Roles Activation” and “Specific Semantic Roles Activation” sections.

5.5.Semantic type-driven activation

A final important aspect of populating the image schematic activation graph is constituted by the semantics of entity types. For example, FrameNet semantic types Lateral, Leftish, and Motion_based_orientation activate Center_Periphery IS, while the frame element Goal in frames like Attaching, Body_movement or Bringing has the FrameNet semantic type Goal, which activates the Goal spatial primitive.

Further examples are provided in Section 7. The queries used for semantic type activation assertions include an initial query listing all existing semantic types, followed by a manual exploration of their differences and coverage, resulting in a selection of semantic types activating some IS or SP. Then, a second query is performed, looking for entities filtered by the aforementioned iteration of non-disambiguated lexical units from synsets and their hyponyms, also extracting their semantic type, ending in a final coherence checking between the entities retrieved, their semantic type, and their semantic type activation of an IS or SP. The queries are available at Appendix A, in paragraph “Semantic Type Activation Query”.

6.1.Evaluation setting

The evaluation setting uses an excerpt of the ISCAT dataset99 [40], and state-of-the-art tools for frame detection from natural language. The ISCAT excerpt has been taken from a cleaned version1010 of the ISCAT online resource. ISCAT is a repository of image schema sentences taken from a large variety of original sources, mainly from literature (e.g. [29,42]), but also from some online sources (e.g. MetaNet, newspaper articles), which are listed in the cleaned repository. The sentences from the excerpt were manually annotated with one IS per sentence.

In this evaluation run, we selected 99 out of 2,478 sample sentences from the cleaned ISCAT excerpt. The main reasons for this extreme reduction of the evaluation set is due to the fact that the sample size per image schema varied considerably and the original dataset only annotates one image schema per sentence. The gold standard is therefore limited to this unique annotation, but image schemas often co-occur in a single sentence or even phrase, and we were interested in whether the image schemas resulting from the evaluation pipeline would at all be possible for the sentence at hand. For that reason, we had to manually analyze all results, so providing a customised manual evaluation in addition to the automated standard evaluation.

To avoid the introduction of bias, further criteria for selecting the set of sample sentences from the larger cleaned repository were (1) variety of original sources, (2) distribution of image schemas, (3) only image schemas already covered in ImageSchemaNet, (4) mixture of concrete (literal) and abstract (metaphoric) examples, (5) English language only, (6) no syntax linearity restrictions, and (7) no minimum or maximum length of the sentence. In terms of variety of sources, we wanted to ensure that not all samples are derived from the same authors, addressing similar ideas or scenarios.

The evaluation setting uses two frame parsers with entirely different architectures, in order to get a finer assessment of the effect of ImageSchemaNet in the process. The parsers include OpenSesame [59] and FRED [28]. OpenSesame is an end-to-end system focused on frame (and semantic role) detection. Its trained model is based on softmax-margin segmental recurrent neural nets. As with most NLP tools, OpenSesame labels extracted textual segments rather then trying to abstract them as entities and their relations in a knowledge graph. FRED is a hybrid knowledge extraction system with a pipeline including both statistical and rule-based components, aimed at producing RDF and OWL knowledge graphs, with embedded entity linking, word-sense disambiguation, and frame/semantic role detection. The big differences between the two parsers are supposed to make evaluation nuances emerge across parsing paradigms (string-centric vs. entity-centric, informal vs. logical representation).

In order to evaluate ImageSchemaNet, we automatically parse natural language sentences in order to annotate them with frames from FrameNet, and we use these frames to get the activated image schemas as encoded in ImageSchemaNet. We then compare the automated annotations to the manual ones, in order to estimate the accuracy of the process, so providing the first results for explainable image-schema detection in natural language texts. Explainability is granted by the heuristic abstraction applied in ImageSchemaNet and in its usage with the parsers. Example of explainability of IS and SP activation from a FRED graph is provided in Appendix C.

The image schemas covered in Framester and their frequency in the evaluation dataset are reported in Table 1, where we can notice that considerably more examples for Containment were included than for the other image schemas. This distribution was selected to reflect the image schema frequency in the original dataset, with by far fewest examples for Support. Finally, both concrete, i.e., directly relating to a physical or real scenario, and non-physical, i.e., transferring physical aspects to a more abstract scenario, such as MIND AS A CONTAINER, sentences have been selected. The evaluation corpus is available in Appendix D as well as on the ImageSchemaNet GitHub.1111

6.2.Evaluation procedure

In order to measure the coverage of ImageSchemaNet, an initial trigger frame is required, to evaluate whether that frame leads to the correct image schema profile. To this end, we used natural language sentences as initial frame triggers and implemented a two-step pipeline. First, we parse natural language sentences with OpenSesame [79] and FRED [28], which return frames for sentences in the test set. Second, frames are in turn used to query ImageSchemaNet, and identify potentially activated image schema profiles.

To evaluate the final result set from this approach, we first performed automated evaluation (against the original manual IS annotation) applying standard information extraction measures: precision, recall, and weighted F1 score. Each natural language sentence in the evaluation set is annotated with exactly one image schema. However, as discussed in Section 7, and shown in Table 2, multiple image schemas are often co-activated in individual sentences or even phrases, and ImageSchemaNet enables the detection of more than one IS per sentence. For that reason, we have performed a second manual evaluation process to establish whether any returned image schema(s) is plausible for a given sentence.

6.3.Evaluation results

The dataset described in Section 6.1 and listed in Appendix D was used to test our pipeline approach for correct linking between frames detected in natural language and underlying image schema. In Table 2 we present data about IS and SP activation from the selected corpus, noting a better performance from FRED except for the IS type, which was limited by default by the current ImageSchemaNet coverage of six image schemas. In Table 3 we present weighted F1 scores for each frame parser as well as for their confusion matrices in Fig. 1 and Fig. 2.

Fig. 1.

Confusion matrix for true and predicted image schema labels using OpenSesame.

Fig. 2.

Confusion matrix for true and predicted image schema labels using FRED.

Table 3 compares the final results of the two parsers on the evaluation set of 99 sentences, where the last column represents the number of sentences which were actually processed. In fact, due to the brevity of sentences, or the metaphorical language or non-linear syntax, in some cases no graph/parsing is produced, resulting in blank result sets, and consequently no image schema detection is possible.

For several sentences, the results of both parsers lead to a co-location of image schemas, i.e., the result set contained more than one image schema, which we counted as correct if the set contained the correct image schema. Please be reminded that multiple image schemas might be correct for a single sentence, and no inconsistency or incompatibility could come from co-location of different image schemas in the same sentence, even if each sentence in this dataset is manually annotated with only one. For the sentences where the result set of the parser contained multiple image schemas, 11 result sets contained the correct manually annotated image schema when using OpenSesame and 15 result sets when using FRED. The overall results of FRED are significantly higher than those of OpenSesame, due to the fact that the former identifies more frames and synsets activating a correct image schema. This is also reflected by the absolute counts of sentences for which frames activating some IS (at least one) were detected, which are 75 for FRED and only 53 for OpenSesame, as shown in last column of Table 2. Table 2 shows that FRED individuates 124 occurrences of frames activating some IS, for 43 different frame types, while OpenSesame activates 57 frame occurrences for 15 types. Note that this number is not the total number of evoked frames, but only those that activate some IS. There are 126 IS activation occurrences (tokens) for FRED and 78 for OpenSesame, while both parsers retrieve at least one IS from each of the six types covered in ImageSchemaNet. Spatial Primitives are mainly activated by verb arguments in OpenSesame: 11 tokens for 2 types, both SP of Source_Path_Goal; and by WordNet synsets in FRED: 20 tokens for 6 types, from Containment, Source_Path_Goal, Part_Whole and Center_Periphery. Tables with IS and SP activation by both parsers are available on the ImageSchemaNet Github,1212 jointly with the FRED knowledge graphs for all the processed sentences.

A confusion matrix is provided for the results derived from each parser. Figure 1 for OpenSesame and Fig. 2 for FRED provide the true labels on the vertical axis and the predicted labels on the horizontal axis. Zero confusion is represented as white space for a clearer visualization and there is no true label for the class “NO_IS” since all sentences in the dataset were annotated with one image schema, however, this class is used to show when no image schema could be returned from the pipeline. When using OpenSesame to detect frames and derive IS (Fig. 1), a total of 94 examples are represented, since 5 sentences lead to a result set of more than one image schema that did not contain the true label. NO_IS therefore means that no image schema was returned from the detection process (46 sentences with OpenSesame). The most confusing image schema is Containment, possibly since, as shown in Table 1, it is the most common in the corpus. For 11 sentences, frames linking to Center_Periphery were returned, e.g. Locative_relation for the sentence There was passion in her eyes. Overall, a tendency to confuse other image schemas for Center_Periphery can be observed. A possible explanation for this tendency comes from the correlation between the semantic extension of these broad IS, and their activators in ImageSchemaNet. Apparently, the semantics of Containment, Center_Periphery and Source_Path_Goal overlaps in examples like He stepped in the middle of a difficult situation: “difficult situation” can be conceived as a Container for the agent; as an area, in which the agent lies in its Center; or as a destination of the agent as a moving entity along a Source_Path_Goal. This simultaneous activation leads to IS combination (cf. [38]) like Crossing_Opening, Going_in or Going_Through.

When using FRED to detect frames and derive IS, the highest number of correct results is obtained for Source_Path_Goal (Fig. 2). However, there was understandable confusion across this image schema and Containment, when frames, such as Motion, were returned, e.g. for the sentence The whole situation spiraled out of control. This confusion is understandable, since Containment and Source_Path_Goal are frequently collocated as movements in and out of a Container. Spatial primitives are activated by WordNet synsets: 76 tokens for 45 types; and FrameNet frame elements: 15 tokens for 8 types activating Source_Path_Goal and Part_Whole; all of them are core frame elements. Out of 99, only 91 samples are represented in this confusion matrix, because the remaining 8 sentences lead to a result set of more than one image schema that does not contain the true label.

Even though there is room for improvement, as shown in Table 3 these results show that this idea of interlinking frames with an image-schematic layer in Framester is promising. The main bottleneck at the moment is the frame parser. For instance, a strong preposition to frame detection component in FRED could drastically improve these results, since prepositions are currently considered in the tool only as features to detect or generate roles, however, they provide a very strong indicator for spatial language and type of image schema (see also [25]).

The evaluation dataset is available in Appendix D, while the OpenSesame parsing file, FRED knowledge graphs generated from text, and manual IS and SP detection files can be found at the ImageSchemaNet GitHub.1313

We have manually inspected the returned image schemas with respect to whether (a) the returned image schema that does not correspond to the original gold standard label could be correct, and (b) whether several returned image schemas actually apply to the sentence at hand. For instance for (a), the expression We are approaching the end of the year is labeled with Center_Periphery, however, clearly shows a collocation with Source_Path_Goal. And for (b), for instance, My symptoms went away is labeled as Center_Periphery. FRED parser, as shown in Fig. 3, detects three frames: Motion, Travel and Departing. All of them activate Source_Path_Goal but Departing also activates Center_Periphery, which is the label from the ISCAT repository. OpenSesame, on the contrary, as shown in Fig. 4, detects only Motion from the verb go, but recognizes the Motion frame element Distance, which has a :coreSPActivation towards Path and Periphery. Consequently, the :IS_Profile according both to FRED and OpenSesame shows a co-activation of Source_Path_Goal and Center_Periphery.

Fig. 3.

FRED graph with image schemas activation for my symptoms went away.

Fig. 4.

OpenSesame graph with image schemas activation for my symptoms went away.

The method based on OpenSesame showed a preference for Center_Periphery irrespective of the gold standard label. In 40 of 53 annotated sequences the method returned one image schema, of which 13 were correct and 16 returned reasonable image schemas even though not corresponding to the gold standard label. The remaining 13 sequences of 53 were annotated with more than one image schema, out of which 10 contained the gold standard label, and 9 out of the 13 represented correct image schema collocations.

The method based on FRED frequently returned Source_Path_Goal when the label is Containment, where in all of these six cases a collocation of both could be observed, e.g. Try to get out of those commitments. Interestingly, slight lexical variations would result in the same set of image schemas, e.g. He took the problem apart piece by piece and He tore the problem apart looking for its solution would both be annotated with Blockage, Source_Path_Goal, and Part_Whole, whereas the gold label only considered Part_Whole. For FRED, 44 natural language sequences out of 65 annotated only resulted in one image schema, of which 24 corresponded to the gold label and 13 where image schemas that can be considered also correct. In 21 cases the method relying on FRED returned more than one label, of which 14 contained the gold standard label and 11 provided reasonable collocations of up to three image schemas.