-----------WORK IN PROGRESS------WORK IN PROGRESS------
Title: INSIDE THE NARRATIVE
Authors: Gustav Kvaal, Torkell Bernsen, Simon Gilbertson
Artistic work/ I SKYGGEN AV KRIG/ SHADOW OF WAR (NO)
Torkell Bernsen – Visual storytelling, Animation, Compositing
Gustav Kvaal – Visual storytelling, Illustration, Animation
Odd Torleiv Furnes – Music and VR-Sound
Rafal Hanzl – Unreal Engine facilitator and Realtime Effects
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
ABSTRACT:
The aim of this artistic research project is to create a VR documentary experience that narrates the story of a time witness from the second world war in Bodø, Norway. The project explores questions concerning visual storytelling and ethics in the encounter between the VR-audience, interviewed subjects and the audiovisual spatial design. Artistic and qualitative research methods have been employed to explore how different visual modes and contexts alter the experience of narrator and narrative in a media form characterized by its ability to place the viewer in a state of immersion, intimacy, and a sense of presence. Theoretically, this study is situated in an artistic landscape connected to media theory, journalism, ethics and visual communication. Concepts such as postmemory, media witness ethics, with the so-called risk of improper distance and considerations around the term distant others, are relevant for the reflection associated with the project. The study proposes a view on ethics towards VR- storytelling that is opening up a the idea having a spectrum of ethical considerations, based on totally immersive VR-simulations to VR-experiences that is extended screen experiences similar to screen-technology today. It is also suggesting a mindset to designing for virtual reality that is more open for utilizing opportunities in the technology than neccesarily creating worlds for escapism.
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
INTRODUCTION:
The artistic research project, Inside the Narrative, derives its name and focus from the narrative point of view of the audience exploring VR-experiences. It curiously joins in on the ongoing exploration of storytelling for immersive media and particularly virtual reality as a storytelling platform. What happens to storytelling when the audience is experiencing the story from within, immersed in the storys own world as audiovisual surroundings? The familiar rectangular framing of books and films is now not ther to guide us in understanding the story. This project has its academic position in the subject area of visual communication at the Faculty of Arts, Music, and Design at the University of Bergen, Norway. The project's position within professional fields such as illustration, animation and graphic design gives us a unique opportunity to help bring forth new perspectives and voices in VR storytelling that may differ from insights gathered from, for example the computer games industry, film, medicine or engineering. The VR-technology has been around for many decades but it is only in the last decade that theory concering storytelling has started to emerge. This project belongs to this emerging field of artistic research. The Inside the Narrative project aim to provide both students and professionals within the field of illustration, graphic design a platform to start developing experiences, knowledge, skills and work methods towards designing for VR-experiences. As a part of this professionalism and knowledge is understanding the media technology and it effects on the audience and stakeholders in the production processes. In a recent artistic project called Shadow of War, a non-fiction VR-documentary became an opportunity to investigate the ethics of VR-storytelling.
VR-technology is known for creating a strong sense of presence for the users in virtual reality experiences (…). The feeling of presence also in non-fiction VR-experiences have amongst other aspects prompted the well known notion of VR as an “empathy machine” that say something about how VR as a medium of communication have a potential power that goes beyond former media viewing situations like watching the news on a screen. The feeling of presence seperates this medium from others and challenges debate on what producers of media content for VR-experiences should consider in the production process. This production process opens up for new possibilities of storytelling but also new challenges.
When approaching a discussion of storytelling ethics concerned with the experience of VR-mediaproduction it has been important for us in this study to to make a clear statement about our understanding of what type of human experience a VR-experience is at this point in time. The differrent views about how impactful, convincing or overwhelming the immersive VR-experience is will stear us in different directions on how to understand an ethical discussion. As a way to describe the different perspectives one might say that we are discussing wether a VR-experience makes us lose the distinction between what is real and what is virtual. A world of VR-experiences that will blur in society of what is real and what is virtual very different measures is needed in comparison to viewing VR-experiences as just a new extension to former audio visual screen technology (Janet H. Murray 2020). As a way to move from traditional media formats of film and print the project is funded in a viewing of VR-experiences as a new arena for multiple types of experiences, not only the fully immersive, realistic real world simulations. We address the discussion about ethics and VR-technology in storytelling based on this as being just another medium of representation, possibly with a higher degree of emotional impact based on the feeling of presence in the VR-environment
--------------------
---------
.
AN EARLY VR- PROTOTYPE: OFF-PISTE 2019
At an eight-day multidisciplinary VR storytelling workshop at Hafjell near Lillehammer in 2019, called Off-Piste Virtual Reality Storytelling Lab, the storytelling itself was put into focus. No Matter Where is the title of an experimental prototype of a VR documentary where you get to meet with various characters and their personal stories in a VR space. The project explores how to obtain, process and present documentary material in the VR format. With this prototype, we wanted to find out what it would be like to be face-to-face with someone who openly shared their life experiences? We meet three different characters within this VR installation. The thematic focus of what is being told is the various individuals' relationship to places and how these have had an impact on memories and their lives.
Wearing VR goggles with headphones in this prototype immerses the viewer into a misty and snowy landscape. The sound of a gentle breeze enhances the experience of the place's presence. The attention of the audience is directed towards a person sitting in the landscape, by means of an enticing sound. As you focus your attention on the person, the person stands up and begins to tell a story. While the person is talking, the audience can move around the space to look at the interview subject from different angles. You can also choose to look away. When you get close to the figure talking, you see in this prototype that the human figure is just a kind of 3-dimensional hollow shell that lacks the backside. This deficiency is related to the volumetric filming technique used to generate the 3-dimensional figure in the VR space. After about 3 minutes, the person ends the monologue and sits down in a waiting position. With the help of another enticing sound, the audience is led on to a new story told by another character in the same space. This time it's the sound of a radio we hear. When you turn around, another character in the landscape is appearing, which in this case is given a kitchen environment that is located in the middle of the misty landscape. As you approach, this person gets up and starts a monologue in the same manner as the first one. In this prototype, each of the stories and sounds that lead the attention is triggered by keystrokes from a technician in the room who observes the progress and course of action of the VR experience. The use of a technician in this project is a possible way to run an event course in a VR experience. The advantage of this solution is that the experience here is personalized to each viewer based on the viewer's body language and attention. This could also be automated to make production less resource-intensive and more easily divisible.
As an approach towards documentary content in this project, we mainly wanted to relate to the real world and photorealism as a starting point for the visual expression. Environmental descriptive elements were therefore gathered with photographic techniques which were later processed into 3D elements and assembled in the game engine called Unity. The snowy and misty landscape backdrop was created by a 360-degree panorama so that we could fill the whole world around the audience. This could just as well have been a 360 video recording if that had been desirable. To create the ground and the place where you stand and where the interview subjects are located, a technique called photogrammetry was used. This technique creates digital 3-dimensional objects by allowing software to analyze depth ratios in the images based on a plethora of photographs taken systematically of a place or object. This way, one can quickly capture and integrate photorealistic objects into a 3D program without necessarily being a 3D specialist.
To give the audience the opportunity to feel that they were in the same room as the interview subjects, we wanted them to be able to move with some freedom in the VR space. Therefore, this prototype does not use a classic 360 video or similar to create a VR environment. With 360 film you get a photographic visual expression, but without being able to move around in a 3-dimensional environment. One is locked to a point in the universe. Instead, the audience is offered an area of 16 m2 where they can move freely and look around both in front and behind the various objects in the space.
To film the interview subjects that we later encounter in the VR experience, we use a technique called volumetric capture. This is done by using a Kinect camera and software called Holo Cap that combines a video recording with 3D data read by an infrared camera. In this way, we could capture a 3-dimensional figure in the space and not only a regular flat 2D video image. It is only from a partially frontal perspective that it works visually from the point of view of the audience. Still, it is not flat as a video recording, so that the figure's body shape becomes visible in space, making it possible to integrate the figure with other 3-dimensional elements in the space.
The sound recordings used in the VR experience were organized digitally in different virtual places in the VR space in the same way as you would distribute 3D objects. In this way, you can organize sound in the game engine Unity (alternatively the game engine Unreal Engine) so that you experience the sound coming from different directions and at different distances.
Through working with the prototype, we engaged in processes and techniques that are largely about creating virtual places with different opportunities for experiences and actions. The audience's role in this particular VR experience was as a kind of mobile observer without so much opportunity to influence the environment. Still, the audience´s meeting with the interviewed subjects and their personal story inside the VR space seemed to be one of apparent emotional impact. The choice of controlling the experience by the use of a technician tells us that VR not only needs to be a lonely digital experience but can also be conceived as a type of performance where the experience is shaped through a feedback loop like in the theater. After working with this VR-prototype, the overall experience is that focusing on the quality of the story itself still remains important also from the inside of the narrative.
