Post-cinematic spectatorship in virtual reality: Negotiating 3DoF and 6DoF in ‘Queerskins: Ark’
Queerkins: Ark is the second chapter of a four-chapter cinematic virtual reality experience Queerskins (2018-ongoing). The piece, made by Illya Szilak and Cyril Tsiboulski in collaboration with the choreographer Brandon Powers, premiered at the Venice VR Expanded exhibition of the Mostra Internazionale d’Arte Cinematografica La Biennale di Venezia that took place digitally – in virtual reality – from 2-12 September 2020.[1] As visitors, wearing a VR headset, enter the virtual world of the Venice Biennale, they embark on a gondola that takes them to the Exhibition Hall (see Figures 1 and 2). From there the viewer can access and watch all the 360-degree videos that have been curated as well as previews of so-called ‘room-scaled experiences’, which could only be accessed in full on commercial VR platforms such as Viveport and Oculus, albeit free of charge for the duration of the festival. While room-scaled experiences and 360-degree videos use two different types of filming and viewing technologies, both need a VR headset to be experienced as intended (more or less sophisticated respectively), and are commonly and often indistinguishably referred to as virtual reality. These two techniques lead to a different involvement of the viewer. 360-degree films give viewers three degrees of freedom (commonly abbreviated 3DoF) as the headset detects rotational movements of the head along the x, y, and z axes and adapts the content viewed accordingly. Building up on this technology, the headset needed for room-scaled experiences also adjusts to the viewers’ vertical, lateral, and horizontal movements in space, offering thus six degrees of freedom (6DoF), which are obtained when the creator uses 3D modelling techniques such as photogrammetry and volumetric video.
For being a relatively novel modality of viewing films, VR tends to be treated in scholarship from a perspective of design rather than as a narrative text. As Mandy Rose writes, phenomenology and experience design may have more relevance than textual analysis to address the specific forms of experience of VR.[2] When one writes about VR, it is in fact impossible to keep away from talking about its apparatus – affordances and glitches alike – since it has a direct impact on the aesthetic and kinaesthetic experience that unfolds from its use. Ark uses both techniques described earlier, coupling moments of 3DoF with 6DoF, and provides the viewer with a varying experience of space and characters. As the piece uses advanced functionalities of VR (6DoF), it required viewing with a higher-end setup than most 360-degree videos (3DoF). With the inability to experience the piece in a dedicated room and with the equipment of the Venice Biennale in 2020 as Fux described in a previous issue of NECSUS,[3] I had to see the piece with my own VR headset linked to a borrowed gaming computer. There is certainly something to be said about the limitations and lack of accessibility of these viewing conditions. On the one hand, headsets and gaming computers are expensive and so VR works offering 6DoF remain largely inaccessible for many, thereby contradicting the principle of portability of digital media. On the other hand, even with a competent setup, I experienced glitches and failures while viewing Ark and concluded that the gaming laptop was not powerful enough to unlock all the possibilities that Illya Szilak mentioned as being possible in an email exchange we had afterwards, such as flying above the characters and defeating gravity if I moved my arms up and down. There is no user manual for Ark or button that one can press to receive instructions. These kinds of motion must therefore arise from intuition, an intuition that gamers may have perhaps developed in greater measure than cinematic spectators.
The first chapter of Queerskins, A Love Story, mostly placed the viewer in the position of a static cinematic spectator, with the unique ability of handling some particular objects. In A Love Story, the viewer sits at the back of a car and gets introduced to the protagonist Sebastian, who died of AIDS, through manipulating some of his belongings packed in a box next to the viewer at the rear of the car. The viewer can thus read passages from Sebastian’s diary, which along with the religious speech sounding on the radio and the conversation taking place between Sebastian’s father, who is driving the car, and mother, who sits in the passenger seat, establishes the circumstances in which Sebastian died: rejected by his family (his father in particular) and their local community in rural Missouri because of his homosexuality.
While the story of Sebastian and his lover Alex may evoke previous cinematic and literary works, the second chapter of Queerskins and its narrative significance emerges from its strategic use of different modalities of spectatorship in VR. Queerskins: Ark starts where A Love Story ended, with the mother, Mary-Helen, unpacking the box that was sitting at the back of the car. After bringing the box up to Sebastian’s childhood bedroom in the attic, she sits and reads entries from his diary. As we hear Sebastian’s intimate memories told in voiceover, the camera slowly moves towards and out the window. The image of the attic blurs and cross-fades on a beach where Sebastian and Alex meet, we learn, for their anniversary. We first hear them talk, and then their intimate encounter progressively turns into a dance between the two characters, first on the beach and then in a completely dark space, in which only the bodies of the characters made of colourful dots remain (see Figures 3 and 4). Ark then concludes in the attic with the mother closing the diary, standing and looking at herself in the mirror as she undresses and slightly touches her body before leaving the room (see Figure 5).
As the mother starts reading in the attic, the viewer witnesses the scene as an invisible cinematic spectator with a 360-degree perspective. Our invisible presence in the space produces an uncanny feeling of being there but kept at a distance. Although we have a head-mounted display (or headset) that allows us to feel ‘present’ in the space, the filming technique only allows us 3DoF and thus our physical movement in the room, except for that of our head, has no repercussion on what we see. In other words, besides being able to tilt or rotate our head and change our angle of vision, we cannot experience the depth of the virtual space. Such aspects of cinematic VR leave viewers on the awkward border between games and cinema, interaction and spectatorship, which may lead to a sense of helplessness and a frustrating viewing experience. Just as a game, the 360-degree video allows us to enter a new environment, but unlike a game it does not allow us to interact with it. To borrow Janet Murray’s metaphor, 360-degree videos place us in a swimming pool but do not let us swim.[4]
While the viewer remains static in A Love Story and at the beginning of Ark, Ark succeeds at creating an immersive atmosphere in alignment with its narrative precisely because it capitalises on, rather than dissimulate, the limitations of VR technology. Similarly, some examples of non-fiction 360-degree videos have marked audiences in the last decade, such as 6×9[5], Notes on Blindness[6], or Travelling While Black[7]. In Travelling While Black and to a further extreme in 6×9, the inability to move within the virtual world make us feel rather than only see and hear what the characters tell us about their own limited mobility.[8] In addition to their playing on our lack of capacity to move, all these pieces including Ark use sound and oral storytelling to immerse us in a sensuous world anchored in reality. This happens through both storytelling in voiceover, which creates a sense of historical authenticity, and binaural sound technology, giving materiality to the virtual space.
In Notes on Blindness, the dark setting and enhanced ambient sounds transmit the personal experience of a blind man while also restricting the visual distraction one may experience when entering VR. In his video-essay ‘Why VR does not promote empathy’, Dirk Eitzen describes that viewers experience too much distraction when entering a new space, which they want to visually appropriate.[9] Citing as an example the 360-degree video The Displaced[10], Eitzen argues that the absence of frame of VR videos, allowing viewers to first explore the exotic space and then listen to the boy’s story, distracts them from the cruel reality they are supposed to witness and hinders their empathy.[11] VR documentaries that reproduce a direct cinema style and focus especially on mediating exotic or astounding images may indeed create major distraction from the storyline.[12] In contrast, the dark and sometimes abstract images in Notes on Blindness draw our attention onto the many ambient sounds, thereby communicating aesthetically what it may feel like to be blind.
VR works that bring major attention to sound, oral storytelling, and gestures, as it is the case of Ark, enhance the audience’s sense of bodily presence and engagement. The audience gains a sense of ‘place illusion’, which Mel Slater defines as ‘the strong illusion of being in a place in spite of the sure knowledge that you are not there’.[13] The two modalities of VR – 360-degree videos (affording 3DoF) and room-scale interactive experiences (6DoF) – convey very different place illusions by either refusing or enabling our habitation and movement within the virtual space.
Ark belongs to the realm of post-cinema and to a sub-genre of VR known as ‘cinematic VR’.[14] While the piece builds on cinematic conventions and engages the spectator’s audio-visual senses, it also introduces some characteristics of games into the cinematic experience as it requires gestures and movements from the viewer to make the narrative experience complete. At first, the 3DoF scene in the attic at the beginning of Ark does not enable the viewer to move within the virtual world, which makes us feel similar to how Sebastian felt stuck in Missouri. Later, when the volumetric filming of the beach gives viewers 6DoF, viewers acquire the physical ability to move in the virtual space (through physically moving in their real environment of viewing). We can therefore choose to move closer to or further from the characters, thereby hearing their dialogue with more or less intensity to the point that the dialogue can become inaudible and be thoroughly covered by the sound of the sea if we move too far away to hear it (see Figure 3). This use of sound encourages our feeling of presence with the characters in the virtual world.
While cinema engages the viewers’ senses of sight and hearing to stimulate other senses, such as the sense of touch,[15] VR offers spatial experiences that engage the viewers’ body kinesthetically, converting us into performers within the narrative. As the encounter between Sebastian and his lover Alex becomes more intimate, the beach fades and turns into a black environment in which the protagonists dance together. The characters move and leave trails of light after their passage, that turn around and cross the viewer’s body, involving us physically in their dance (see Figure 4). This mise-en-scène plays with the paradoxical relation of VR with the body of the viewer. If we come in the virtual world embodied – that is, with all our senses active and ready to inhabit the new virtual environment – our body is not represented in the virtual world, which produces an uncanny feeling of becoming a performer without a body. As the characters’ vibrating particles of light come intimately close to us or seem to cross our invisible body, however, so do the lights somehow compose us and give us a sort of visible embodiment.
The dancers play between closeness and distance to the viewers’ body (that is, to the camera), stimulating our sense of touch and gestures from our part. While the representation of the beach and the attic activated our visual sense, the darkness displaces the visual focus and allows our involvement in the virtual world and connection with the characters through a multiplicity of senses. The dancers swirling around draws us into their movement, causing us to move closer as they grow distant from us, move into their light and feel that we are part of their movement and that we also respond to the music. As Matthew Reason and Dee Reynolds theorise the ‘kinaesthetic empathy’ of spectators of live dance performances, they write that the movement of others on one’s ‘body, imagination and feelings’ provoke ‘sympathetic, empathetic, or contagious’ effect.[16] In a sense, Ark produce an experience that may be closer to live performance than to cinema because of how it includes us in the same physical space as the performers by rupturing the invisible fourth wall separating audiences from performers and appealing to our motion and gestures.
As Ark calls for movement and gestures that situate us within the same space as Sebastian and Alex, it invites us to imagine the world differently. Sebastian’s dialogue with Alex on the beach places emphasis on his inability to properly come out to his conservative Christian family and his will to find a home elsewhere. The beach and the dark space offer queer spaces that the characters can call home. The gestures expected of and made by the viewer towards the characters create a connection, a bridge between our bodies and that of the dancers. As the NECSUS issue on #Gesture explores, gestures in media practice can become a way of knowing, of creating bridges between two instances, and of giving form to ideas.[17] By inviting our gestural involvement in the virtual world, Ark stimulates and questions our knowing of queer realities and of places we call home.
Alex’s and Sebastian’s body take form as assemblages of colourful dots reminding of the pointillism painting technique. This aesthetically and narratively gives shape to an incompleteness (see Figure 6), as products of a memory described in the diary – and thus necessarily incomplete – and as queer people existing on the margins of a heteronormative American society and partially invisible to public discourse (see Figure 7). This bodily appearance results from Szilak and Tsiboulski’s use of the limitations of VR to their advantage,[18] as they explain: ‘although this lack of realism was necessitated by the limitations of the volumetric technology itself, it worked well with our narrative’.[19] Our gestures in the virtual world become ways of joining the characters in their incomplete states. As we move into the light of the characters’ incomplete bodies, we somehow feel that we embody the characters themselves or provide them with a physical reality, a sense of completeness.
In contrast to cinema or live performance, VR works made of volumetric video such as Ark ask us to move and interact with the environment for the narrative to unfold akin to a game. Through our movement, we thereby activate a voiceover of the mother’s and Sebastian’s voice speaking out excerpts from the diary:
We only had one rule. Alex could not bring his lovers home. Home meant something to me in a way that it didn’t to him. Then, he broke the rule.
Our bodily action in the virtual environment thus grounds the dance taking place around us in a personal queer narrative.[20] While these textual passages were used by the choreographer Brandon Powers to choreograph the dance of the characters, hearing them let us come closer narratively speaking to the queer space the characters created for themselves away from Sebastian’s conservative family environment. In a sense, gestures make us part of the medium itself. As they connect us to characters, texts and images, gestures renew our body awareness not only in the virtual world but also potentially beyond it.[21]
Through gestures, VR artworks have the potential to create a kinaesthetic empathy, exchanges that affect our orientation into the world. Contact with others shape how we orient ourselves towards determinate situations and beings, that is, how we perceive ourselves and others around us, and how we move into the world. Sara Ahmed describes orientation as taking a direction, a direction that has been made available to us, similarly to how sexual orientation arises from a socio-cultural delimitation of who is made available as objects of love and desire.[22] By requiring our gestural presence in its aesthetic and narrative world, Ark aims to transform our orientation in the world, or rather make visible and available other paths that often remain invisible.
As Queerskins: Ark uses different modes of filming and interaction with the audience, it situates its VR experience in between gaming, live performance, and film viewing. When the dance completes, the viewer is taken back to the bedroom in the attic where the mother finishes reading the diary, stands in front of the mirror, and looks at her female body as she undresses. When she hears the voice of her husband downstairs (Sebastian’s father who appeared in the first chapter of Queerskins), she swiftly leaves the room, which closes the experience. By inviting the viewer into their trails of lights, the characters make available a world of queer lines, the same queer lines that bring the mother to touch her body in front of the mirror before leaving and take awareness of her own sexuality (see Figure 5). It in fact remains unclear whether the memory of love between Sebastian and Alex takes shape as a dance because of how the mother imagines it or as a poetic authorial representation of an intimate moment. Through this closing, Ark shows how the re-enactment of her son’s memory re-oriented the mother’s world and led her to question her relation to her own body and her previously heteronormative orientation towards others.
In the attic with the mother, the viewer is taken back to a 360-degree cinematic viewing experience and a realist representation. This realism, along with the off-screen voice of the father, calls upon the viewer’s familiarity with cinematic conventions, and replaces us in an environment that may seem more familiar than the imaginary worlds on the beach and in the dark space. This change of environment and mode of interaction – from 360-degree video (3DoF) to volumetric video (6DoF) and back to 360-degree video – create a clear connection between the fictional world and our physical environment. Ark brings us on a journey that aesthetically and phenomenologically blends the boundaries between the real and the virtual and between modalities of mediated experiences.
Maud Ceuterick (University of Bergen)
Acknowledgments
Research for this article was conducted with funding from the European Union’s Horizon 2020 Marie Skłodowska-Curie Action research and innovation programme under grant agreement No. 800259.
References
Ahmed, S. The cultural politics of emotion. Edinburgh: Edinburgh University Press, 2004.
Ceuterick, M. ‘Queering space through technology: Re-writings of collective memory in AR and VR’, Alphaville, vol. 21, 2021; Forthcoming.
Ceuterick, M. and Ingraham, C. ‘Immersive storytelling and affective ethnography in virtual reality’, Review of Communication, vol. 21, no. 1, 2021: 9-22; DOI:10.1080/15358593.2021.1881610.
de Rosa, M. ‘On gesture, or of the blissful promise’, NECSUS. European Journal of Media Studies, vol. 8, no. 2, Autumn 2019: 113-128; DOI: https://doi.org/10.25969/mediarep/13141.
de Rosa, M. and Hediger, V. ‘Post-what? post-when? a conversation on the “posts” of post-media and post-cinema’, Cine´ma et cie, vol. 16, no. 9, 2016: 26-27.
Eitzen, D. ‘Why VR does not promote empathy’, Journal of Videographic Film & Moving Image Studies, vol. 7, no. 2, 2020: http://mediacommons.org/intransition/why- vr-does-not-promote-empathy.
Ex Nihilo, ARTE France, and Archer’s Mark (prod.). Notes on blindness [Virtual Reality], 2016.
Fux, N. ‘VR on the film festival circuit: IDFA & IFFR 2017-2019’, NECSUS_European Journal of Media Studies, vol. 9, no. 1, Spring 2020: 307-314; DOI: https://doi.org/10.25969/mediarep/14319.
Poulton, L. and Panetta, F. 6×9 [Virtual Reality], The Guardian, 2016.
Fuchs, P. Virtual reality headsets: A theoretical and pragmatic approach. London: CRC Press, 2017.
Grønstad, A. Film and the ethical imagination. London: Palgrave Macmillan, 2016.
Marks, L. The skin of the film: Intercultural cinema, embodiment, and the senses. Durham-London: Duke University Press, 2000.
Murray, J. Hamlet on the holodeck: The future of narrative in cyberspace. Cambridge: MIT Press, 1997.
Nash, K. ‘Virtual reality witness: exploring the ethics of mediated presence’, Studies in Documentary Film, vol. 12, no. 2, 2018: 119-131; DOI: 10.1080/17503280. 2017.1340796.
Reason, M. and Reynolds, D. ‘Kinesthesia, empathy, and related pleasures: An inquiry into audience experiences of watching dance’, Dance Research Journal, vol. 42, no. 2, 2010: 49-75.
Rose, M. ‘Technologies of seeing and technologies of corporeality: Currents in nonfiction virtual reality’, World Records, vol. 1, no. 1, 2018: 1-11; https://vols.worldrecordsjournal.org.
Slater, M. ‘Place illusion and plausibility can lead to realistic behaviour in im-mersive virtual environments’, Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 364, no. 1535, 2009: 3549-3557.
Sobchack, S. Carnal thoughts: Embodiment and moving image culture. Berkeley-Los Angeles-London: University of California Press, 2004.
Solomon, B. and Ismail, I. The displaced [Virtual Reality], The New York Times, 2015.
Szilak, I. and Tsiboulski, C. Queerskins: A love story [Virtual Reality], Cloudred, 2018.
_____. Queerskins: Ark. [Virtual Reality], Cloudred, 2020.
Williams, R. Traveling while black [Virtual Reality], Paul and Felix Studio, 2018.
[1] Program available here: https://www.labiennale.org/en/cinema/2020/venice–vr–expanded
[2] Rose 2018, p. 2.
[3] Fux 2020.
[4] Janet Murray defines immersion as a ‘metaphorical term derived from the physical experience of being submerged in water’, it is about ‘the sensation of being surrounded by a completely other reality. . . that takes over all of our attention, our whole perceptual apparatus’, such that participation in this new reality involves learning ‘to do the things that the new environment makes possible’ (1997, pp. 10; 98-99).
[5] Poulton & Panetta 2016.
[6] Ex Nihilo, ARTE France and Archer’s Mark 2016.
[7] Williams 2018.
[8] See Ceuterick & Ingraham 2021.
[9] Eitzen 2020.
[10] Solomon & Ismail 2015.
[11] About empathy in VR documentaries, Kate Nash writes that they may risk to place the audience at an ‘improper distance’ from the subject, to form appropriate moral and contextualised responses to an other’s situation (2018, p. 129).
[12] Visual distraction in VR also emerges from the low resolution, latency, or glitches that images in VR often have compared to cinematic or televisual images – especially when viewed with a low-end headset. This is also reinforced by the grid marking the boundaries of the virtual space that comes up if one tries to move beyond it, and the occasional, but much too frequent, positioning of objects too close to the viewer, thereby ignoring the principles of common human visual field and acuity.On this topic see Fuchs 2017.
[13] Slater 2009, p. 3551.
[14] On post-cinema see De Rosa & Hediger 2016.
[15] About haptic images see for example Marks 2000 and Sobchack 2004.
[16] Reason & Reynolds 2010, p. 72.
[17] de Rosa 2019, p. 115.
[18] In the case of VR, the limitations of the technology often give birth to productive narrative elements. For example, Vestige (Bradbury, 2018) and Travelling While Black (Williams, 2018) use both the affordances and limitations of the medium to create embodied experiences of memory in between forgetting and remembering. While Vestige, made with 3D technology, uses trails of light instead of fully fleshed bodies to represent dead and mournful characters, Travelling While Black draws a narrative correspondence between the technical limitations of 360-degree video that leave the viewer invisible and unable to move, with the many restrictions on movement and existence that African-Americans endure in the United States. Their success as acclaimed award-winning pieces can certainly be attributed to the link they develop between the VR technology and the embodied and cultural process of individual remembering (see also Ceuterick & Ingraham 2021).
[19] Shared with the author in a private exchange
[20] See Ceuterick 2021.
[21] See Grønstad 2016, p. 158, on medialised gestures in the work of Giorgio Agamben and Pasi Väliaho.
[22] Ahmed 2004, p. 11; 94-95. When talking about VR, at the Electronic Organization Conference‘s ‘A workshop in VR about VR’ (2020) that I co-organised with Jill Walker Rettberg, Illya Szilak mentioned Ahmed’s book Queer Phenomenology as an inspiration for her work.