1. Concept
1.1. Introduction
The concept of presence in Embodied/Encoded is concerned with the physical and embodied dimensions of musicking in relation to various digital-informational representations of such phenomena. I explore presence through practices of recording, creative coding, audio production, and spatial electroacoustic music composition. Technologies associated with these practices offer a window into the physical dimensions of sound. They provide information streams tightly coupled with physical or perceptual features—a starting point for composing links between material abstracted from the physical world and digital processes underlying their representation. Digital abstraction renders sound a flexible, malleable entity—a ‘flickering’ signifier to borrow from Hayles1—capable of reconfiguring presence in creative ways.
Of particular interest is the link between presence and the sensual/corporeal aspects of sound production and listening. Research in embodied music cognition and extended reality provide a foundation for this mode of meaning formation. In virtual reality studies, for example:
the term presence has referred to a widely reported sensation experienced during the use of virtual reality specifically, but also found during the use of other media. Users experiencing presence report having a compelling sense of being in a mediated space other than where their physical body is located.2
In the context of immersive electronic music, I suggest that physical presence – a sense of ‘being somewhere’—emerges not just from images/representations of place, but from peripheral aspects of the (embodied) acoustic experience such as spatial proximity and distance, diffuseness, resonance and reverberation, noise floor, etc. Consequently, musical meaning in “Embodied/Encoded” moves away from the symbolic dimension of electronic sounds toward meaning as an outcome of embodied interaction with the environment. The concept of presence can also apply to ‘mixed’ music, or combinations of acoustic and electronic sources, in which virtual presence is complicated by real bodies in real spaces.
Interfacing with presence is a digital-informational ontology characterizedby stasis, quantization, and replication—metaphors or technical descriptions of the digital medium generating another gravity in the reconfiguration of presence. The dance between encoded and embodied dimensions of sound inspires and informs this artistic research. Through a body of immersive music and multimedia documentation, Embodied/Encoded traces connections between presence and its digital abstractions, exploring meaning as a dynamic oscillation between human and non-human influence.
This research was awarded with resources connected to Natasha Barrett's Reconfiguring the Landscape (RCL) project. In addition to Natasha's supervision and training in technologies, composition, and idea development, my own research work found interesting connections to the RCL project, detailed in section 1.5, “Site Unspecific.”
1.2. Extended Reality
Biocca provides an account of how the concept of presence was formulated in virtual reality (VR), originating with the term telepresence. Eventually shortened to presence, the concept was “generalized to the illusion of ‘being there’ whether or not ‘there’ exists in physical space.”
This generalization … allowed theorists of presence to include the fantasy environments of narrative and game designers as well as the abstract iconic representations of scientific visualization.3
The field, along with the concept, has expanded to technologies of augmented reality (AR) and mixed reality (MR), all of which, as explained by Lee, “are encompassed in the umbrella term extended reality (XR).”4 Of note is that the longest established academic journal devoted to extended reality is called Presence. I draw from Biocca’s VR studies because of his unpacking of the notion of ‘being there’ and its adaptability to concerns of electronic music, particularly the composition and experience of virtual spaces. Electronic music, however, has different medium-specific concerns than XR and offers different possibilities for presence to manifest: those emerging from sound and performance without layers of visual simulation.
Biocca describes three kinds of presence pertinent to the XR experience: physical presence – a sense of ‘being there,’ social presence – being with another, and self presence – a sense of self/body/identity. These work together as users navigate and interact with virtual environments. Interaction is key, and where much electroacoustic music (my own included) deviates from VR’s paradigm of presence. My music does not require listeners to interact with the sound such that their movements and perspectival shifts change the auditory scene. Particularly for my compositions which fall into the acousmatic5 genre, I am interested in physical presence—the sense of being somewhere, evoked with the aid of technology and compositional actions to be described in context of specific works. My interest is not only in the representation of real spaces, but also in the creative reconfiguration of physical presence and the effect on meaning formation. Two guiding questions toward this end are:
How closely does the technologically mediated aural experience mirror our acoustic experience in the physical world?
What does the mediated aural experience afford in the context of artistic expression?
Other dimensions of presence are relevant in contexts involving live performers. Even in the acousmatic scenario, social presence comes into play as listeners observe sounds composed or performed with intention by another person. I will return to social presence in my discussion of collaborative work with live performers.
1.3. Embodied Music Cognition
Self presence may seem irrelevant to the musical listening experience because listeners’ bodies are not supplanted by virtual ones. However, the embodied experience of music is central to this research and it is through embodied meaning formation that self presence can be viewed as essential to acousmatic listening. Research in embodied music cognition explicates how certain patterns in sound are actually ‘encoded’ in our bodies due to the reflexive coupling between embodied subjects and their environments. Leman explains:
Rather than considering a listener, say, as a mind that receives input (music) and produces output (e.g., descriptions of perceived emotions or dance movements), the embodied way of understanding considers a listener in a closed interacting loop with her or his musical environment. The loop is constrained by the human body, hence ‘embodied.’ It is assumed that human musical action and perception are reciprocal processes that fuel that loop, and that action and prediction are co-determined by constraints of the musical environment, as well as by those of the (corporeal) organism that interacts within it.6
In this sense, presence originates in the listener and at a level of cognition distinct from conceptual or language-based modes of signification.
Of particular interest is the concept of musical affordance, which points to structural “meaningful” features in music that potentially match with humans’ interaction capabilities. Musical affordances, such as metrical structure, can therefore be conceived as extended meanings contained in structures outside the human brain. They facilitate embodied interactions, such as precise synchronization, and they guide attention and awareness of musical events at higher levels of meaning formation.7
Here we see that ‘interaction’ in music has as much to do with potential action responses as it does with actual action responses. Musical meaning formation is layered, and these ‘action’ responses are not confined to movement, but extend to the other senses as well. As Leman summarizes from Broeckx’s theory, “physical properties of musical sound such as frequency, spectral density, and loudness give rise to impressions of visual and tactile space, such as extension, density, weight, smoothness, roughness, hardness, softness, liquidity, ephemerality, flatness, consonance, dissonance, and other impressions.”
Broeckx thereby asserts that these assessments are not merely metaphors. … In line with the theory of embodiment mentioned here, assessments draw upon synesthetic and kinesthetic experiences, such as spatial orientation, material density, auditive coloring, and different degrees of momentum of sounds.8
Cross-modal associations have been investigated further by De Poli et al. (2017) and referred to as sensorimotor expressiveness:
Sensorimotor expressiveness refers to aspects not covered by musical and emotional expressiveness, since it investigates the domain of cross-modal associations. The starting point is that metaphorical descriptions may offer possibilities to explain and understand aspects of the musical experience otherwise ineffable.9
These details shift focus from movement (i.e. dance and rhythm) in action-oriented ontology to sound properties often peripheral to music, which may more closely define the holistic acoustic experience. They may include various spatial properties related to proximity, echo, resonance, and reverberation and ephemeral details in transients or noise floor. They may be details embedded in complex sound fields such as the soundscape, where sounds always happen in the context of other sounds.
1.4. Post-Schaefferian Acousmatic
Sensorimotor expressiveness can be heard/felt in much post-Schaefferian acousmatic music—particularly that influenced by Denis Smalley’s writings on spectromorphology in which the author establishes a music/sound semiology around archetypes of the acoustic experience. A hallmark of post-Schaefferian acousmatic music is the use of anecdotal recorded sounds of instruments, non-musical objects, or environments coupled with the creative use of sound processing and synthesis tools which transform sources, sometimes beyond recognition. Smalley’s concept of indicative fields proposes links between “human experience and the listener’s apprehension of sounding materials in musical contexts.”10 Such links can be formed even in the absence of recognizable sound sources. On the subject of gesture, Smalley writes:
When we hear spectromorphologies we detect the humanity behind them by deducing gestural activity, referring back through gesture to proprioceptive and psychological experience in general.11
In “Space-form and the acousmatic image,” Smalley devotes a section to Transmodal Perception: “although acousmatic music may be received via a single sensory mode, this does not mean that the other senses lie dormant; in fact they spill over into sonic experience.”12 Embodied cognition and sensorimotor expressiveness reinforce Smalley’s insights, in a sense confirming what we know intuitively from the acoustic experience. In the context of this research, Smalley demonstrates how archetypes of the acoustic real world can be observed and composed in acousmatic virtual worlds.
Natasha Barrett’s work in the post-Schaefferian lineage has provided a model for technical methods in the domain of spatial audio and an artistic example of visceral music which unites physical presence with high corporeal engagement. In the article “Spatio-musical composition strategies,” she describes cross-modal experience in relation to spatiality and memory:
Although we principally use sight, all our senses interact and support each other. For example, our judgement of speed would be inaccurate without aural cues. Hearing can also reveal spatial information when the eyes are useless: in the dark, when encountering a visual barrier, or when spatial information is outside our field of vision. Changes in temperature, humidity, touch and smell also contribute to our spatial awareness and spatial memory. When we re-visit spaces from our past it is the interaction of information from all our senses that evokes the memory – not only the immediacy of our vision.13
1.5 Site-Unspecific (on field recording and representation)
Our bodies and imaginations engage in sonic transcription and reproduction more than the machines we have invented for these purposes.
—Francisco Lopez, “Profound Listening and Environmental Sound Matter”
A discussion which has resurfaced throughout this artistic research in dialogues about my music (and by extension others in post-Schaefferian lineage) is the status of the field-recording in relation to issues of representation, ethnography, and ethics.14
As Demers summarizes: “Sound can transmit information about space, place, and location. And depending on the music and the type of listening approach, the same sound can bear different messages about site.”15 Soundscape composition diverges aesthetically from post-Schaefferian acousmatic by foregrounding the context/situatedness of sounds in the environment. Field recordings are often treated as documentary evidence for ecological or ethnographic study. In Joanna Demers’s definition:
Soundscape works can feature purely natural sounds or those of machines or other forms of technology; human voices can be present or absent, depending on the work. Although soundscape compositions might also contain newly composed material, the use of audio footage stakes a claim on objective representation, a quality usually perceived as lacking in material that is entirely composed or improvised. Audio footage ties a soundscape composition to the ecological, social, historical, or cultural dynamics of a specific location, which both personalizes and politicizes the act of listening.16
R. Murray Schafer was central to the development of the field of ‘acoustic ecology’ introducing the concept of soundscape and establishing the World Soundscape Project in the 1960s. From the outset, the movement distinguished itself from the abstract or de-contextualized treatment of environmental sound recordings in acousmatic music. Westerkamp emphasizes: “Acoustic ecology or soundscape studies – the study of the interrelationship between sound, nature and society – is the arena from which this work and thus, the term soundscape composition emerged in the first place in the mid-1970s, and it is that arena that gave it its context its voice and its strength.”17
The acousmatic works of Embodied/Encoded consist, in part, of modified and raw field recordings made in Oslo, Norway with other sites regularly interjecting. In order to position myself in relation to soundscape composition at recent research forums, I have traced the evolution of my source recordings as they move between states of site-specificity and musical abstraction in a given project. I explain that my post-production encounter with these materials leaves a different kind of emotional imprint than what I experience in situ. My attention while recording is usually focused on technical aspects such as the mic levels, placements, and wind noise. I am also focused on my own body movements and sounds, all of which take my attention away from the present acoustic experience. What I rediscover and reimagine in the studio context is the visceral, sensual essence of this sound matter (to borrow from Francisco López).18 It can be a surprising encounter as there is sometimes little correlation between the phenomenal experience of sound in the environment and the same sound experienced in recorded format in the studio. Mundane recording sites can yield rich and beautiful results while grand, premeditated schemes sometimes yield unflattering ones; the reverse is also true. I gravitate toward recordings which seem to offer possibilities for musical intervention in which I amplify my post-production impressions, thus reconfiguring the VR trope of ‘being there,’ suggesting site-specificity, to being somewhere … site unspecific.
In this artistic research, I wanted to engage the world in my immediate vicinity while reacting to a more global experience of chaos and order. I view the methodology as a rejection of commercial practices of electronic music production, due in part to the idiosyncratic specification of how sounds and technology merge and to what musical ends. In another way, it challenges traditional notions of musical pre-conception—the act of specifying or staging particular actions or objects a priori—because materials were derived from incidental or improvisatory scenarios and sculpted later in a post-production context. Channeling Xenakis, this mode of composition felt like extracting patterns from chaos through reflexive acts of listening and computing, even though the end result is mostly determined.
My interventions can have the effect of removing site-specificity, emerging in its place residue of the digital process which served to destabilize the image in the first place, pushing the sound to a different mode of representation. Oscillation across sonic archetypes of acoustic realism and digital/synthetic is one way I enact an interplay between embodied and encoded dimensions. Some methodologies are described in the sections that follow.
Sound artist Francisco López’s work often consists of field recordings presented in raw states, without any post-production or editing. Despite our different methods, López articulates a similar attitude toward representation in field recording. He writes of his work La Selva:
La Selva (the musical piece) is not a representation of La Selva (the reserve in Costa Rica). While it certainly contains elements that can be understood as representational, the musical piece is rooted not in a documentary approach but in a notion of ‘sound matter.’ …
What I’m defending here is the transcendental dimension of the sound matter itself. In my conception, sound recording does not document or represent a richer and more significant ‘real’ world. Rather, it focuses on the inner world of sounds. When the representational/relational level is emphasized, sounds acquire a restricted meaning or a goal, and this inner world is dissipated.19
Digital simulation artist Ian Cheng creates living autonomous worlds and explores agents’ capacities to deal with these ever-changing environments.20 His comments on representation also resonate:
For me, representation is a sideshow to the attempt to make an artwork feel alive. … The aliveness of an artwork is the most important quality to me right now because it is only when we are confronted with living things that we can cognitively look beyond what they represent or symbolize, forgive their contradictions, and begin to see their underlying complexity. It would be a sign of critical retardation to look upon a living dog for example, and only see it as a symbol. No, it is a living being, and it forces us to confront all its habits, misbehaviors, history, roles, and accept seeing all this mess at once. What if an artwork could reliably open up this way of seeing? I am obsessed with this possibility.21
The real-world, site-specific installations of Natasha Barrett's Reconfiguring the Landscape project additionaly influenced my investigation of virtual spaces. Early in the fellowship period, Natasha involved me in her experiments exploring the concept of presence and interaction in the urban "soundscape/soundspace/landscape,” with a focus on the outdoor, public space context and content. Specific moments included experimental work for the Subliminal Throwback installation located in the amphitheatre between Gydas vei 8 and Slemdalsveiein 11 at Majorstuen, Oslo and the installation Presence/Nærvær which was located at Akershus Fortress. The RCL installations can be described as augmented reality, placed in the real-world, while my work became an antithesis of this in the virtual world, placed in the concert zone, and emphasising the notion of site unspecific. The influence of XR in both projects resulted in an aversion to strongly gestural archetypes characteristic of post-Shaefferian work (see section 3.2. Still Life).
In summary, the aesthetic intentions of Embodied/Encoded are distinct from acoustic ecology and soundscape composition while sharing similar materials and methods. Site-specific materials remain intact at many moments, but this layer of meaning is not central to the musical or corporeal meaning. Still, I acknowledge the site-specificities of my recordings and have attended to them by documenting sound sources of my compositions on their respective pages, including location, date, source, and recording equipment used. While not a prerequisite for listening, this information does anchor the works in material, causal origins. The recordings offer rich listening experiences on their own terms.
1 N. Katherine Hayles, How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics (Chicago: University of Chicago Press, 1999), 30-35.
2 Frank Biocca, “The Cyborg’s Dilemma: Progressive Embodiment in Virtual Environments,” Journal of Computer Mediated Communication 3, No. 2 (September 1997): 12.
3 Ibid., 12.
4 Hyunkook Lee, “A Conceptual Model of Immersive Experience in Extended Reality,” PsyArXiv (September 13, 2020): 2, doi:10.31234/osf.io/sefkh.
5 For convenience, I borrow the definition of acousmatic from Bayle to “demarcate music on a fixed medium—representing a wide aesthetic spectrum—from all other contemporary music.” F. Bayle, Musique acousmatique: propositions … positions (Paris: Buchet/Chastel, 1993): 18.
6 Marc Leman, Micheline Lesaffre, and Pieter-Jan Maes, eds. Embodied Music Interaction (New York: Routledge, 2017), 1.
7 Ibid., 2.
8 Ibid., 15-16.
9 Giovanni De Poli et al., “Beyond Emotion: Multi-Sensory Responses to Musical Expression,” in Embodied Music Interaction, ed. Marc Leman, Micheline Lesaffre, and Pieter-Jan Maes (New York: Routledge, 2017), 79.
10 Denis Smalley, “The Listening Imagination: Listening in the Electroacoustic Era,” Contemporary Music Review 13, No. 2 (1996): 83.
11 Denis Smalley, “Spectromorphology: explaining sound-shapes,” Organised Sound 2, no. 2 (1997): 111.
12 Denis Smalley, “Space-form and the acousmatic image,” Organised Sound 12, no. 1 (2007): 39.
13 Natasha Barrett, “Spatio-musical composition strategies,” Organised Sound 7, no. 3 (2002): 313.
14 These have taken place at seminars and forums organized by the Norwegian Research Council in Artistic Development Work.
15 Joanna Demers, Listening Through the Noise: The Aesthetics of Experimental Electronic Music (New York: Oxford University Press, 2010), 115.
16 Ibid., 120.
17 Hildegard Westerkamp, “Linking soundscape composition and acoustic ecology,” Organised Sound 7, no. 1 (2002): 52.
18 López prefers the term “matter” to Schaeffer’s “object” because “it better reflects the continuity of the sonic material one finds in sound environments, a continuity affirmed by the non-representational approach to sound recording.” Francisco López, “Profound Listening and Environmental Sound Matter,” in Audio Culture: Readings in Modern Music, ed. Christoph Cox and Daniel Warner (New York: Continuum, 2009), 85.
19 López, “Profound Listening,” 85.
20 Ian Cheng, “faq,” Ian Cheng artist website, http://iancheng.com/faq (accessed October 31, 2023).
21 Ian Cheng and Hito Steyerl, “Simulated Subjects: Glass Bead in conversation with Ian Cheng and Hito Steyerl,” Glass Bead Journal – Site 1. Logic Gate: The Politics of the Artifactual Mind (2017), https://www.glass-bead.org/article/simulated-subjects/?lang=enview (accessed October 25, 2023).