Ontology supported semantic based image retrieval

2.1.Image retrieval

In the field of Image Retrieval, fundamental studies have focused on Text-based Image Retrieval and Content-based Image Retrieval approaches. In a study conducted in this field [26], a comprehensive research was conducted on feature extraction (color, shape, texture) from images, system architecture, and proposed solutions to the encountered problems. In the Text-Based Image Retrieval approach, the desired results cannot be obtained in search operations due to the mismatch caused by the semantic difference between the content of images and keywords. The main reason for this is the inability to fully express the meaning of the image with the label information (Tag) used to describe the image. The fundamental method used in the text-based document retrieval approach is expressed as Latent Semantic Indexing (LSI). In this method, clustering is performed to identify semantically similar words. In the search process, not only the words matching the keywords but also the words in the same cluster are searched. In a study [28], the LSI method was used in accessing web documents along with the image attributes found in the document. In the Content-Based Image Retrieval approach, the fundamental method used in search operations is defined as Query-by-visual-example (QBVE). In this method, low-level features (such as color, shape, texture) are extracted from the image used for querying, and as a result of the search process, images matching these features are retrieved from the database. In a study in this field [16], a method referred to as Query-by-semantic-example (QBSE) is employed, where in addition to image features, concept vectors describing the image are also utilized. With the development of web applications, there has been an increased necessity to identify image content on the web. Tag structures are used for describing image content. In many cases, the label information used to describe image content may lead to the use of inappropriate labels due to people’s different perceptions of images. In a study [11], an image similarity graph was used in addition to image features. In a research study [4] aimed at tackling the issue of Semantic Gap, which emerges between low-level features (such as color, shape, and texture) and the high-level semantic concepts perceived by humans, an ontology-based model was developed. This model was designed to define objects and their relationships identified from satellite images. Subsequently, the created model was applied in the process of searching for similar satellite images. In our study, we propose a workflow that can be regarded as a more comprehensive approach to the process of Semantic-Based Image Retrieval.

2.2.Object detection

The initial phase of image inference involves the Object Detection process, where Computer Vision systems employ Deep Learning approaches. In a conducted study [29], the examination of performance results centered on Object Detection architectures (Region Proposal, Feature Pyramids, CNN), application areas (Salient object detection, Face detection, Pedestrian detection), and various datasets (PascalVOC, Microsoft COCO). This analysis was carried out by employing Deep Learning approaches. In another study [6], comprehensive details are provided on the performance of Object Detection utilizing the Fast Region-based Convolutional Network (Fast-RCNN) architecture across different datasets (Pascal VOC, Microsoft COCO). YOLO (You Only Look Once) is one the preferred object detection system in object detection processes. With YOLO, object detection can be performed over 9000 object categories [17]. In this study, Object Detection processes were performed using the Detectron211 framework developed by Facebook Research and based on the Faster-RCNN [20] method.

2.3.Visual relationship detection

In visual relationship detection, the goal is to determine an expression that describes the relationship between two objects detected in an image. The process of visual relationship detection involves extracting semantic meaning from the image. There are three distinct tasks in visual relationship detection. Performance results for each of these three tasks, namely Phrase Detection, Relationship Detection, and Predicate Detection/Recognition, have been examined in various studies [10,23,24,27]. The semantic inference of an image is performed using N detected objects in the image and K predicates that describe the relationships between objects. In this case, the complexity of learning all possible relationships is expressed as O(N2K), and obtaining training data examples for all possible relationships is challenging. In the process of extracting meaning from an image, it is envisioned to only consider a useful number of relationships [2]. Defining the relationship between two objects is related to their interaction with each other (Spatial Relationships). This interaction is expressed with a predicate that defines the relationship between two objects, such as a spatial position (on, under) or an action (holds, moves). In a study in this field [15], the relationship between objects was represented in the form of subject-predicate-object using visual and spatial features of the objects. In the first stage of the Semantic-Based Image Retrieval process proposed in this study, Predicate Detection was performed referring to a conducted study [9].

2.4.Scene representation

To derive meaning from an image, it is essential to represent objects and their relationships using an appropriate method. In a conducted study [22], objects and their relationships were modeled using the Scene Graph method. In the Semantic Web approach, Ontologies are used to model concepts and establish relationships between them. In a general description, Ontologies are utilized to provide a clear and formal representation of concepts within a specific domain. In a study conducted on ontologies developed for multimedia applications [5], a new schema named Multimedia Web Ontology Language (MOWL) has been proposed for the representation of multimedia content. With the proposed schema, the term “concepts” is used to represent concepts related to real-world entities (Real World Entities), and “media objects” is used for the representation of multimedia concepts. In the second stage of this study, the representation of objects and their relationships was accomplished using Ontologies.

2.5.Semantic-based image retrieval

The Semantic-Based Image Retrieval approach is used to address the Semantic Gap problem. Low-level features (color, shape, texture) extracted from regions identified in the image through Object Detection are transformed into ontologies. This transformation aims to convert these features into a structure that computers can process and derive meaning from. In a study [12], an Object Ontology is created from low-level features, and keywords are defined for objects. The defined keywords are used in the representation of high-level concepts, and the search process is conducted based on these keywords. In the Ontology-Based Image Retrieval system created for nature images [18], the process involves searching for semantically similar images using query sentences created by users. In another study [14], an image retrieval system is proposed for the domain of the Asteroidae flower using an ontology that defines classes, subclasses, and properties. Whereas, in another study [7], the created Scene Graph was utilized in the process of searching for images that have a similar structure. In this study, a two-stage approach for Semantic-Based Image Retrieval using Ontologies has been proposed to address the Semantic Gap problem.

2.6.Similarity measurement

In this study, it is proposed to use the Entropy method to measure the Semantic Gap, which is the fundamental problem encountered in the Semantic-Based Image Retrieval approach. In information theory, the number of bits required to encode data that will be stored, transmitted, or compressed between computers is determined by the Entropy method developed by Claude E. Shannon [19]. In a study [3], a four-step process for measuring the Semantic Gap has been proposed. In the first step, image features were defined with ontologies. The color, size, texture, and location features of the image were divided into four parts, and an object ontology was created. In the second step, combinations of image features were defined, and using the Entropy method, the amount of information contained in the image was measured. In the third step, combinations of image features for query purposes were defined by users, and again using the Entropy method, the amount of information was measured. In the fourth step, the Semantic Gap measurement was made by examining the relationship between the measured amounts of information obtained from image features. In another study [21], the proposed Content-Based Image Retrieval (CBIR) approach involves measuring the similarity between two images using the Graph Matching method. Additionally, a language model was developed to describe the similarities between two images in written form. In the measurement of the Semantic Gap, in thıs study suggested to use the Joint Entropy method to calculate the number of similar relationships between two images.

3.1.Dataset

To establish the relationship between two objects, Bi-RNN was employed for training the created model, and subsequently, it was used in the search operation. The Visual Genome dataset played a crucial role in generating the ontologies for this purpose. In this study, following essential data filtering and transformation processes, a total of 59,724 defined association samples were selected from the first 5,000 images within the Visual Genome dataset. The data selection was based on 100 object categories and 70 predicate categories, referencing a previous study [10]. For the training and testing phases of the model, the dataset was divided into 75% for training, 15% for validation, and 10% for testing data. The details regarding the selected data from the dataset, the training results of the model, and the processes of Ontology creation will be explicated in the relevant sections.

Fig. 3.

Modified Bi-RNN model.

3.2.Word embedding

Word Embedding is employed in Natural Language Processing (NLP) applications to represent and demonstrate the semantic similarities between words. Each word is converted into a numerical vector using the word embedding method. Therefore, In our study, when delineating the relationship between two objects, the class labels of the objects (e.g., “person,” “motorcycle”) were converted into a 100-dimensional vector through the utilization of the word embedding method. Words with semantic relationships are positioned closely to each other in the vector space. In this study, the word embedding process was executed through the application of the Continuous Bag of Words (CBOW) method [13] within the Word2Vec algorithm. The Gensim22 library was employed for creating word vectors in this study. During the training stage of the Word2Vec algorithm, the data in the subject-predicate-object structure selected from the Visual Genome dataset, specific to our problem, were utilized. An example of the data selected from the dataset and employed in training the Word2Vec algorithm is as follows:

3.3.Model

In this study, a redesigned Bi-RNN model, was used to describe the relationship between two objects. In the Bi-RNN architecture depicted in Fig. 3, the vector Xt represents the input sequence at time t, ht denotes the hidden layer, and Y represents the output data corresponding to the input. When determining the outcome information at time t, Bi-RNN utilizes information from both the previous and next nodes in the sequence.

The Bi-RNN model performs the computation process in both the forward and backward directions in the hidden layer. In visual relationship detection processes, the arrangement of subject and object information in the expressed relationship <subject-predicate-object> affects the generated outcome (predicate). Therefore, the ordering of subject and object information in relationships such as <person-on-motorcycle> and <motorcycle-on-person> leads to different results. The reason for using the Bi-RNN model is its ability to learn the differences arising from the ordering of subject and object in relationships expressed in the <subject-predicate-object> form.

The model was designed according to Eq. (1). The first term of the equation represents the forward sequence of information, while the second term represents the backward sequence. The model takes the subject, spatial information, and object data [x1, x2, x3] as input. The output, displayed by Y=RK equation, provides the distribution generated for a total of K predicates selected for the first 5000 images.

A Bi-RNN computes the hidden states twice: a forward sequence h1→ and a backward sequence h1← where Wh1y→ denotes the input-hidden weight matrix in forward direction.

During the training stage of the model, the selected data from the Visual Genome dataset was used. The data was arranged in a specific order and utilized as the input for the model. In this sequence, x1 represented the subject vector, x2 denoted spatial information, and x3 represented the object vector data. The representation of the relationship (including spatial information) between two objects was determined in accordance with Eq. (2).

In the Eq. (2), the x and y correspond to the center coordinates of the bounding box of an object, whereas w and h signify the width and height of the bounding box respectively, that encodes their relative spatial relationship which is important for relationship representation.

In this study, the predicate detection process was executed. In this process, the class labels associated with two objects and the location information in the image, referred to as Subject and Object, were provided. The aim was to determine the expression defining the relationship between them. This process can be represented using Eq. (3):

In the Eq. (3), the expression Pi(O1) gives the probability that the object detected from the image belongs to class i, whereas, the expression Pj(O2) gives the probability that the object belongs to class j in the Object Detection process. On the other hand, Bi-RNN (O1,O2,Z) shows the expression that determines the predicate and defines the relationship between two objects.

The developed model was trained on the Turkish National e-Science e-Infrastructure (TRUBA)33 provided by TÜBİTAK ULAKBİM High Performance and Grid Computing Center.

In this study, the Recall@K metric was utilized for the performance measurement of the proposed Visual Relationship Detection process in the first stage. The Recall@K performance is calculated according to the Eq. (4), providing the probability of the ground truth being among the first K predictions (Top-K Predicate).

The reason for choosing this metric is the selection of the Predicate Detection task in the proposed first stage of our study, where Visual Relationship Detection is employed. Predicate Detection aims to determine an expression (predicate) that defines the relationship between two objects. In this case, when evaluating the model’s performance, we examined the cases where the ground truth is among the first five predictions as determined.

3.4.Ontology creation

Ontologies serve the purpose of describing entities by creating classes within a specific domain and establishing relations between these classes. In this study, an ontology was generated, as depicted in Fig. 4, utilizing the data structured in the subject-predicate-object format from the Visual Genome dataset. These ontologies were employed in the process of Semantic-Based Image Retrieval, and the outcomes of the search are detailed in the relevant section.

Fig. 4.

Relation ontology.

4.1.Meaning inference from the image

During the testing phase of the model on an image not present in the training dataset, objects within the image were initially detected using Detectron2. Subsequently, a predicate was established to define the relationship between the two objects by generating dual combinations of the identified objects. The results obtained from this process are illustrated in Fig. 5.

Fig. 5.

Meaning inference from the image.

4.2.Searching process using ontologies

Meaning inference was achieved by determining a predicate that describes the relationship between objects perceived from an image. The relationship structured in the subject-predicate-object format was then transformed into ontologies and employed in the search for semantically similar images. The outcome of the search for semantically similar images using ontologies is depicted in Fig. 6.

Fig. 6.

Searching process using ontology.

4.3.Measuring the semantic gap

The current study suggests measuring the Semantic Gap, a primary challenge in Semantic-Based Image Retrieval, by calculating the quantity of similar relationships between two images using entropy. The Entropy method, used to measure the amount of information in Information Theory, is utilized for measuring the Semantic Gap between two images. The entropy method generally expresses the disorder of a system. An increase in disorder within the system will lead to an increase in the entropy value. When the entropy of the difference between the numbers of relationships in two images is high, it indicates that there is a high semantic similarity between the two images. Conversely, when the entropy of the difference between the numbers of relationships is low, it can be understood that the semantic similarity is low.

Joint entropy is a concept that measures the uncertainty between two or more random variables. The joint entropy equation is used to calculate this uncertainty and is as described according to Eq. (5):

In the given Eq. (5)

The Semantic Gap between two images was calculated using the Joint Entropy method. This involved utilizing the number of relationships (X) found in the image used for query purposes and the total number of relationships (Y) in the image with similar relationships identified as a result of the query.

According to Eq. (5), as a result of the calculation using the relationship numbers between the two images, three cases were examined:

In this case, according to the Eq. (6), if the entropy (H) is “1” it implies that the number of relationships between the two images is equal, and the two images are semantically similar. If the entropy value is between “0” and “1” it indicates that the number of relationships between the two images is different, and there is a semantic difference between them. When the entropy is “0” it signifies that there is no similar relationship between the two images, and the two images are different from each other.

In the search process utilizing the sample image illustrated in Fig. 7, for the objects “apple” and “bowl” detected from the image, a similar image was found as a result, conforming to the specified “apple-on-bowl” relationship.

Fig. 7.

Searching process using ontology.

As a result of the query, information about the image with a similar relationship is given in Table 2.

Fig. 8.

Result image.

As a result of the search process, the relationship (apple-on-bowl) identified in the image used for query purposes was also detected in another image. The outcome of the Semantic Gap measurement between these two images is provided in Table 3.

According to the entropy-based calculation, out of the six ground truth relationships defined in the image found in the query result, only one relationship matched. Based on the calculation result according to Eq. (5), it was determined that there exists a Semantic Gap.