Developing augmented reality applications to promote digital storytelling: The cases of Choirokoitia and Sandby Borg

1.1Digital storytelling

Digital storytelling expands the trajectories of experiencing and sustaining Cultural Heritage often through enriched, augmented, and immersive user experiences mediated by different technologies and tools, which range from interactive games to extended reality (XR) applications and Artificial Intelligence. It is a dynamic, adaptive, technologically- and culturally-embedded process that can enact affordances for audience engagement, collaboration, creativity, and interaction in digitally-driven contexts (see Psomadaki et al., 2019). Its genesis is attributed to oral culture. As Egan (1989) notes, “But perhaps the most important of all the techniques developed in oral culture is the story” (p. 13). As a closed compound term, the term story follows a specific structure, driven by genre-specific elements and structural components, while the intransitive verb telling “describes the connection between the teller and the audience, but not only. It also involves the space and time in which the telling itself takes place, but also the space and time in which the actual story takes place” (Bratistis, 2022, p. 296). Over the years, digital storytelling has turned into a polysemous term. Bull and Kajder (2004), for instance, view digital storytelling “a series of still images combined with a narrated soundtrack to tell a story” (p. 47). Paolini and Di Blas (2014), on the other hand, define storytelling as more complex and encompassing culturally-embedded term: “a crucial keyword: it implies that the user is not provided just with information or data but that she will get a story or, better, several stories, conveying cultural messages” (p. 34).

The nature of digital storytelling is often contingent upon the different tools and creative ways implemented to engage and involve the audience in different genres and fields, including CH, and goals and objectives of each digital storytelling (see Podara et al., 2021). Digital storytelling is embedded in meaning-making value that often draws on rich cultural and historical practices, values, and traditions. This meaning-making value engages the audience, while interactivity offers a way to navigate through and explore the narrative, especially through the availability of and accessibility to different devices, software, and hardware. As Alexander (2011) succinctly notes, “Digital stories are currently created using nearly every digital device in an ever-growing toolbox” (p. 3). In CH, this constellation of tools and devices can serve as a powerful mechanism for developing cultural material, through interactive audiovisual content that may provide virtual representations of culture, artifacts, and customs, and practices and guide in sustaining cultural values, traditions, and identity. Digital storytelling can unequivocally contribute to the construction and maintenance of one’s identity.

Digital storytelling is the art of experiencing “the power of personal expression” constructed from the narrator’s personal experiences, interpretation, and narration of events (see Bull & Kajder, 2004, p. 47). The scope of digital storytelling is “to provide an enormous array of entertainment experiences” (Miller, 2004, preface). However, this entertaining value can also enclose a deeper salient cultural, historical, and linguistic value within specific community contexts, including engagement and participation and interconnectedness to a community network (see Kim & Ball-Rokeach, 2006). Other scholars, like Alexander (2011) define digital storytelling by using five critical elements: (1) compelling narration, (2) meaningful context, (3) visual elements integration for pathos, (4) music and other materials used to strengthen ideas, and (5) thoughtful and meaningful audience reflection. Taking it a step further, Lambert (2006) views storytelling as a creative and collaborative art driven by seven critical elements: (1) a point of view, (2) a dramatic question, (3) emotional content, (4) the gift of one’s voice, (5) the power of music (soundtrack), (6) economy in audiovisual and written content, and (7) pacing to maintain attention and interest (pp. 45–59). AR can be deployed as a tool to convey a story or in the case of Choirokoitia and Sandby Borg it can serve as a tool to inform users about the historical facts that unfolded during that era and allow them to derive their own conclusions.

2.1Design principles

Drawing on previous scholarly work on DGBL, Kucher (2021) focuses on five underlying design principles that can guide the design, development, and deployment of effective games for pedagogical purposes: interactivity, immersiveness, adaptive problem-solving, feedback, and freedom of exploration. As Kucher (2021) notes, “These principles derived from successful implementation of digital gaming elements in learning, and they provide suggestions and strategies for incorporation of these elements based on real-life examples” (p. 215). The constructive implementation of these principles is neither limited to Virtual Environments nor to formal institutional learning contexts. They can be applied in multiple other areas, including Cultural Heritage and digital storytelling mediated by AR game-based experiences. Interactivity, for instance, can be realized when the game dynamic conditions are set to facilitate meaningful interaction between a particular player or a set of players and the physical and virtual contexts. Interactive content can enhance a player’s level of engagement and participation (see Kucher, 2021; Barab et al., 2007; An & Bonk, 2009). Depending on the nature and design of the DGBL, players can interact with other players, virtual and physical contexts, 3D artifacts, avatars, written content, and other interactive elements and multimodal content (see Aliprantis et al., 2019). Immersiveness can be achieved through DGBL experiences by introducing users to the storyline or historical events and assigning them specific tasks that call for asserting a particular role and taking certain responsibilities. For instance, Holden and Sykes (2011) invited students to interact with non-player characters (NPL) and assert the role of a detective trying to collaboratively solve a mystery by exploring and sharing clues to determine who murdered Dionisio Silva. Their newly assumed identity, collaborative nature of the game, and role within the game immerses users in simulated-real context, where they are driven by a specific goal, to solve a murder.

Adaptive problem solving engages users in challenging problems, with often real-world implications, which guides users to gradually build skill and knowledge. This process may involve several steps such as (1) defining the problem, (2) setting an action plan to solve the problem, (3) implementing the action plan to address the problem, and (4) evaluating the action plan outcome and determining if all available resources were used to successfully address the problem (see Polya, 1988; Kucher, 2021). Any feedback provided during this process may serve as a mechanism to help learners commit to realizing the game objectives, assess their progress, reflect on areas where they successfully apply strategies and areas where they could still improve. Kucher (2021) points out that “In the context of DGBL, feedback may also take the form of accumulation, level of progression, receiving new titles or acquisition of magic objects which provide some visible progress for even relatively small successes” (p. 218). Positive feedback can also be in the form of encouragement or support, offering an indication that a task has been completed and motivating users to proceed to the next and often more complex task. This feedback can be descriptive (e.g., You have just completed all scenario objectives), comparative (e.g., Your score in completing this activity was well above the average score), and evaluative (e.g., Congratulations, you can now progress to the next scenario) (see Kucher, 2021). Finally, freedom of exploration of the DGBL environment, collaboration in identifying new paths or ways to solve a particular challenge that users encounter, and the lack of real-life consequences can motivate students to explore and undertake different approaches or actions to solve a problem (see An & Bonk, 2009; Gee, 2005).

2.2The study

The Digital Methods Platform for Arts and Humanities (DiMPAH) project developed a total of seven novel Open Education Resources (OERs) on digital methods that can be implemented in interdisciplinary contexts. The AR game-mediated scenarios were designed as Open Education Resources as part of the DiMPAH project, between 2021 and 2023. DiMPAH was funded under the Erasmus plus scheme and aimed to aggregate, link, and develop OERs that could be openly accessed on the Dariah Teach platform. OER six, “Design, development and deployment of Augmented Reality Applications,” focuses on the development and deployment of AR game-based scenarios to enrich users’ knowledge and engagement with Cultural Heritage through innovative technologies. As part of the DiMPAH project and OER 6, a series of novel game-based AR scenarios were developed to promote Cultural Heritage and storytelling among Swedish and Cypriot students. The scenarios focused on the Neolithic settlement Choirokoitia, which is located in the southern coast of Cyprus and the iron-age ringfort of Sandby Borg, which is located on the southeastern coast of Öland, Sweden. The landmarks were selected because of their historical significance and the mysteries involving the massacre in Sandby Borg and the disappearance of the Choirokoitia residents. The mystery surrounding both sites was ideal for creating game-based activities to guide users in developing cultural and historical knowledge, engaging in a narrative storytelling, and collectively building knowledge and understanding on each culture and its history.

2.3Methodology

Drawing on the five design principles for DGBL, interactivity, immersiveness, adaptive problem solving, feedback, and freedom of exploration were implemented to design the AR-mediated game-based activities on Choirokoitia and Sandby Borg. A bottom-up approach was adopted where software developers, instructors, and researchers from various disciplines exchanged ideas on possible game-based scenarios to enact interactive, immersive, and collaborative learning experiences entrenched in rich historical, cultural, and linguistic values. As Saif et al. (2021) note, “Lack of user involvement can result in a poorly designed solution, or even a solution that conflicts with user’s needs” (p. 354). The research teams met several times throughout the project to identify the nature of the game-based scenarios, their objectives and goals. Drawing on the mystery involving the abrupt move of residents in Choirokoitia and the massacre at Sandby Borg, the team introduced a game-based framework for users to explore these historical sites, collect information and artifacts, explore clues, and collaboratively solve these mysteries. A total of three scenarios were introduced as well as an introductory scene for each site. The game-based AR scenarios were entertaining but at the same time were entrenched in engaging narration, meaningful context, audiovisual materials, and offered a path for reflection (see Alexander, 2011).

UNITY 3D, with the AR Foundation framework, a novel multiplatform game engine was deployed for the development of the AR game-based activities. The UNITY game engine was selected because it is the most widely implemented platform for developing Android mobile games, especially AR-driven. Further, it is user friendly, features a UNITY Assets Store offering multiple tools and assets, and can engage players across different platforms. The AR applications were developed on the ARToolkit for Android and deployed on mobile devices. The Spatial Localization and Mapping (SLAM) method was used for accurate tracking of the environment and visualization of 3D and 2D objects, such as Choirokoitia circular homes and a Sandby Borg house in users’ physical environment. The SLAM method was also used to render animated characters or avatars, various historical artifacts, green-screen video recordings, speech, music, and sounds, and various triggers, which can be activated when users are at close proximity to a specific object.

3.1Choirokoitia case study

The Choirokoitia AR application featured the Neolithic village of Choirokoitia, which dates between the 7th and 4th BC. This heritage site consists of well-preserved mudbrick and stone circular homes offering a glimpse into the living conditions, social practices, and organization of Choirokoitia residents. The AR scenario was designed to include an introductory scene and three game-based scenarios. The introductory scene: As soon as users startlog into the scene application, they are presented to a Neolithic female character/avatar, speaking about the life at the Choirokoitia of that era, then they are invited to first locate Cyprus on the Mediterranean map world atlas and then the village of Choirokoitia on the map of Cyprus. Initially, users can view Choirokoitia in 360 and as they progress in the scenario they can view Choirokoitia in a 3D format, placing emphasis on the distinctive circular Neolithic homes. In the first scenario, users continue to navigate Choirokoitia in a 3D format, where they are invited to identify the required construction materials to reconstruct the circular structures, stashing them in their virtual inventory. If they successfully complete this activity, users can compose a written report to share this experience. In the second game-based scenario, users come across a young girl who wanders around, desperately searching for her mother. Users need to collect information from the young girl that will guide them in locating her mother. As they reach the village, they discover the young girl’s mother and at the same time, through various activities, explore gender roles in that era. In the third scenario, users explore the materials used during that era to light a fire. They need to collect and determine the specific materials used and their order of use, then place them in the firebase in orderto light the fire through a Point-Of-View angle. Finally, users need to write a reflection on the value of fire and tools used to light the fire during that era. The scenarios were developed and deployed by using the five DGBL principles.

3.2Sandby borg case study

The Swedish case study was based on Sandby Borg, an Iron Age ringfort (c. AD 400–550) situated right by the coast on the island of Öland, Sweden. The structure of the site consists of an oval limestone wall, c. 4 m wide and originally at least 4 or 5 m in height. Three of four gates lead into the interior of the fort, which was occupied by approximately 53 houses, with stone walls and turf roofs. The application was designed to include an introductory scene, with a male avatar of that era providing some basic information about the site and discoveries, as well as a female avatar at the end of the narration. Three game-based scenarios were then developed where users are invited to solve tasks and/or reflect upon the mystery surrounding the site. The first scenario focused on Iron Age food preparation. The task was introduced by a female avatar, explaining that a great feast is about to take place with a lot of guests, so she would need some help preparing the meal. Users are then assigned to search and collect ingredients and utensils to be used during this feast. A number of 2D photos of potential ingredients and cooking utensils, accompanied by simple explanatory texts are distributed, including potential Iron Age articles, as well as food and utensils not likely used in the period in question. Once the relevant objects are collected, users can light the fire and start cooking the meals to be served. The second AR scenario focused on the mystery of the massacre, inviting users to reflect upon the massacre rather than focusing on factual learning. Through narration, users are invited to navigate through the 3D scene and to find various digital objects (3D and 2D) that provide clues about the mystery of the massacre. A female ghost floats freely inviting users to uncover the reasons that led to this massacre. In the third AR scenario, which unfolds in House 40, green screen recordings of a male and female actor in costume, offer a personal narrative and invite users to identify objects in the house that could guide them in identifying the underlying reasons that the massacre took place.

3.3Interactivity and immersiveness

Within the introductory and subsequent three scenarios, interactivity was accomplished with the Choirokoitia village, in both 3D and 360 formats; the circular home structures in 3D; the female and male avatars representing residents of that era; the various artifacts including construction materials, such as mudbricks, stones, and ropes; the collaborative nature of the game-based scenarios; and goal-oriented activities. Meaningful game-oriented activities were developed guiding users in engaging with the 3D Choirokoitia environment, identifying and undertaking the required actions to complete the game and progress to the next scenario. In the introductory scenario, for instance, a female avatar offers users a brief oral historical overview on Choirokoitia, its historical significance, lifestyle, inhabitants, and professional preoccupations. The female avatar was designed to represent a young female figure of that era; the average age for women during that era was only 33. The female avatar was dressed in a simple gray woven wool and flax dress. A 3D circular home structure magically appears in the female avatar’s right hand and rotates around, indicating the significance of these Neolithic home structures. Users are then encouraged to simply tap and activate points of interest, such as locating Cyprus on the map. In other cases, such as the second scenario, direct interaction with the young female avatar is essential to assist the young Choirokoitia girl in finding her mother. In the first and third scenarios, on the other hand, interactivity is achieved through the various activities where users are invited to collect artifacts, such as materials used in the construction of circular homes, including stones, reeds, and bricks, and lighting of a fire, including fungus, a flint stone, and pyrite. Some of the materials were not used during that era, so users need to select the right materials before they proceed to the next level or complete the activities. These goal-oriented activities have a cultural and historical significance because they guide users in finding out critical information on Choirokoitia, its residents, their everyday and professional activities, and other relevant information. The scenario completion calls for the construction of new knowledge and in some instances peer collaboration to realize specific activities, such as locating the young girl’s mother.

Similarly, in the Sandby Borg scenarios, a mixture of 3D-models, 2D images, avatars, greenscreen recordings and sound effects were used to create an interactive and immersive experience. In the first Sandby Borg scenario, the foodstuffs and utensils were in the form of original 2D images accompanied by brief descriptive and informative texts. Alongside the objects relevant to collect (such as herbs, cabbage, wild berries and nuts), other not so relevant objects were also included (such as potatoes, meatballs, plates, knives and forks, jewellery) to guide users in building knowledge on the food ingredients of that era. Similar to Choirokoitia scenarios, users were guided in learning about the period represented by the ringfort, and the completion of the scenario was designed to require users to choose the right objects thus building new knowledge on the Iron Age in Scandinavia. The third Sandby Borg scenario was designed to invite users to search for clues both outside and inside the digitally reconstructed House 40 explaining the massacre. During that process, objects could be collected that refer to actual finds recovered during excavations of that same house. When an object was collected, a greenscreen recording of an actor in costume appeared, explaining the massacre based on the object just collected. In this scenario, interactivity was achieved on a cognitive level, through presenting different conflicting explanations for the mysterious massacre and promoting further reasoning at the user’s end. While there is no “right or wrong” answer concerning this, the setup was designed to encourage users to think critically and derive their own conclusions about the underlying reasons that have led to this massacre. In a collaborative learning setting, this could be elaborated into sharing of ideas and explanations. The information included a description on the livelihood and societal conditions of Iron Age in Scandinavia in general and specific details on the fortress, its appearance today and its anticipated appearance 1500 years ago. All information was based on archaeological evidence and knowledge from recent excavations.

The activities were designed to be collaborative and invite users to assume particular roles, such as investigators trying to solve the mystery of the Sandby Borg massacre and Choirokoitia residents’ abrupt move from the village. These newly assumed roles and identities serve as a catalyst in immersing and engaging users in the game (see Kucher, 2021; Arachchilage et al., 2016; Hadjistassou & Molka-Danielsen, 2016; Holden & Sykes, 2011). The roles, however, are not static or confined to a single identity. Each activity is designed to invite users to assume different roles, based on the activity’s goals and objectives. For instance, users can assist a young girl in finding her mother. In the final Choirokoitia scenario, however, they can take the role of helping to identify the required materials to light a fire, gaining knowledge in this area and at the same time understanding the importance of such practices in Choirokoitia residents’ lives. The multimodal nature of the application, which combines real, physical, and 3D images and artifacts of Choirokoitia and Sandby Borg, and the multisensory context of this digital storytelling enact immersive experiences. The multisensory value is realized through the female narrator’s introduction to Choirokoitia and its cultural and historical significance, the special effects, such as the sudden appearance and rotation of a circular home structure, the avatars, and the multiple artifact used to create a realistic game-driven scenario rich in cultural value (see Kucher, 2021 for a discussion on multisensory representation).

3.4Adaptive problem solving and feedback

The Choirokoitia scenarios were designed to immerse students in game-oriented contexts, embedded in real world problems and implications. The game-based activities introduced several times relevant information on Choirokoitia residents, their lifestyle, and other relevant aspects. At the same time, the activities were designed to start from simple tasks and progress gradually to more complex and demanding activities that require research, knowledge about Choirokoitia cultural and historical practices, values, and lifestyle to complete. The activities also called for building more advanced digital skills. For instance, progressing from the introductory scene where users have to tap on points of interest, the first and the third scenarios require the identification and collection of assets to be placed in users’ inventory to successfully complete the activity. In the third scenario, users need to (i) address the problem of materials needed to light a fire; (ii) determine the best possible solution by identifying the specific materials used to light a fire and rejecting the ones that are not used in this process; (iii) collect and place the materials in their inventory to light the fire; and (iv) determine if they successfully completed this activity by properly using the materials, placing the appropriate artifacts in the fire camp to light the fire. These problem-solving activities help maintain users’ interest, enhance their problem-solving and digital skills, and guide them in completing the tasks by building new knowledge and skills (see Kucher, 2021; Polya, 1988)

Similarly, in the Sandby Borg scenarios, initially in the introductory scene users received some factual information on Iron Age ringfort and the massacre, without being invited to complete any complex tasks. However, as the scenarios progressed, they were designed to be more demanding and engaging, guiding users to search for clues to solve the different tasks. In the second scenario, for instance, the male narrator informs users that after the big feast that they prepared in the first scenario, it is eerily quiet, without any Sandby Borg inhabitants in the ringfort. Users are invited to (i) navigate through Iron Age ringfort to search for clues to find out where the inhabitants have gone; (ii) touch upon specific artifacts that appear in the scene, including a bone comb dropped in the street and scattered broken pottery pieces; (iii) follow arrows or other-relevant clues that lead users into the house where they discover the bodies of six murdered individuals; (iv) assist a female who keeps asking users to help her find her missing brooch, so she can reveal to them the brutal events that led to the massacre. The problem-solving activities are designed to motivate users to assist a person in need in order to receive information that can help them solve the mystery of the massacre.

Critical to this process of activity realization is feedback provision which was designed to offer positive evaluative comments. For instance, in the case of Choirokoitia, when all scenarios goals and objectives are realized, the system was designed to offer written evaluative feedback in the form of notifications: “Well done, you completed all scenario objectives”, or “Congratulations, you can now progress to Scenario 2.” As Kucher (2021) notes constructive evaluative feedback can guide users in building self-efficacy, turning into more confident and autonomous users. Further, neither the Choirokoitia nor the Sandby Borg scenarios have any real-life consequences or game-based consequences, such as being banned from the game. In both scenarios, the consequences were designed to prevent users from progressing to the next level until they successfully complete the indicated tasks. In the case of the second Sandby Borg scenario, users cannot find the truth about the massacre unless they provide the specific piece of jewellery, i.e. the brooch. In the case of Choirokoitia, negative feedback was designed to simply inform users that they had made the wrong choice, inviting them to consider the remaining alternative assets and 3D artefact to determine the specific ones that would help them complete the indicated tasks. Special sound effects and speech recordings offering negative feedback were integrated in the game-based activities to create aesthetic effect on the wrong choices and guide users in reflecting on their answers and identifying alternative solutions based on the available 3D assets and artifacts. In the Sandby Borg food preparation scenario, if users attempted to collect the wrong asset, negative feedback was provided through a text explaining why it was not correct. In both the Choirokoitia and Sandby Borg scenarios, the nature and combination of these game-based elements were designed to maintain students’ level of motivation by engaging and immersing them in the game, rewarding their accomplishments, and guiding them gradually to achieve the required outcomes.

3.5Freedom of exploration

Both the real physical context portrayed through the 360 photos and the 3D format of the Neolithic village and the Iron Age ringfort were designed to promote freedom of exploration, which is essential in not simply becoming acquainted with the two sites but also in exploring the historical events that unfolded during that era and their significance, everyday practices, and cultural value. Users have the freedom to navigate through the two sites, the game-oriented scenarios, and collaborate with peers to complete the activities. During this process, the scenarios were specifically designed to facilitate some flexibility in exploring the socially and historically situated contexts, evaluating the available options, and making mistakes. In the introductory scene of the Choirokoitia scenarios, for example, if users do not identify Cyprus or Choirokoitia on the map, they receive a negative indication (marking turns to red) that they have not identified the right point of interest, so they need to try again until they succeed. Similarly, in the final Choirokoitia scenario, if users do not select the required materials for lighting a fire to place them in their inventory, they cannot place them in the fire camp to light the fire. In the case of the Sandby Borg introductory scenario, students can explore the map of Sweden in order to point to Sandby Borg. The available options include the Swedish cities of Stockholm, Helsingborg and Gothenburg, as well as southeast Öland where Sandby Borg is located. After they identify the exact location, a 3D object of an Iron Age arrowhead appears, which, at a close proximity, can be collected/stashed in the user’s inventory. In the first Sandby Borg scenario, where users are invited to assist in the food preparation, they can first navigate through the 3D site, select the right 3D objects that will help them prepare the different meals. This selection process offers some ingredient options that were not used at the time, including potatoes and meatballs, inviting students to explore the kind of ingredients that Sandby Borg residents used at the time. In both the Choirokoitia and Sandby Borg scenarios, neither the risk nor the consequences are severe, but were rather designed to offer users the freedom to explore the available options and understand each material’s role in lighting the fire during that era. If, on the other hand, users feel that they are not progressing through the game-based scenarios as they anticipated, they can easily opt to exitlog out of the application, restart the scenario, and pursue a different approach to solve the indicated activity, without facing any severe or negative consequences. The scenarios were designed to offer users the flexibility in making mistakes, reflecting upon them, and pursuing a different path to resolve the challenges. Through this flexible approach, users can explore, understand, and undertake the required steps and actions to progress through the game-based scenarios and successfully complete the required activities.