Introduction
Technology has the potential to blur the distinction between composers, performers, and listeners. The diverse musical practices associated with the twentieth century (and beyond) demonstrate this acutely. For example, even the “avant-gardists of the 1960s sought to break down the barriers between composer and audience by such essentially sociological means as non-traditional concert environments and audience participation pieces” (Kramer 1988: 80). It is against the backdrop of this tradition that I, as composer for The Memory Dealer (henceforth referred to as TMD), seek to approach the question of music and its place in immersive drama. In this article, after a brief historical summary in which I contextualize my efforts in relation to particular strands of twentieth-century musical thought, I shall turn my discussion towards more recent research regarding immersion and digital media, drawing particular attention to highly relevant debates concerning audio in video games. Central to this will be a practical account of the process of composition and the eventual realization of the score (and audio generally) in productions of TMD; this will be supported by musical examples that illustrate the potential application of techniques associated with video game and film scoring and will be approached with an eye towards future projects and productions of a similar nature. By way of conclusion, I shall entertain some philosophical thoughts concerning, chiefly, verisimilitude and the difficulties music encounters as an ancillary component within any kind of immersive medium, above all, of a narrative kind. Through all this, I hope ultimately to make the case that music can and should – and indeed, in the case of film and video game already does – help both to influence and to generate new types of narrative immersion.
New directions and non-linear musics
It goes without saying that an interactive or adaptive music is necessarily nonlinear. Musical nonlinearity has been a pre-occupation of composers and music practitioners since long before the technology that might make it a reality existed, or rather, a technology that could, at least, spontaneously account for the decisions and movement (in a physical space) of its listener and create a desired musical experience – still determined – but in an indeterminate environment. In a sense, Messiaen’s much celebrated “end of time,” in which forms “appropriate to a non-linear, non-directed music” which express “new kinds of ‘timeless’ time” (Cross 1998: 61) need not simply be regarded as a concert hall irrelevance in our discussion of dynamic music for a new media today. Indeed, the exploration of this musical stasis, in direct contrast to a properly linear tonal music rooted in the exploration of (and eventual return from) familiar harmonic territories, has been an almost singularly determining factor characterizing art music of the twentieth century (and beyond). As Jonathan Kramer poetically puts it, the epoch of Stravinsky and Debussy is regarded no longer by
the tension-laden pedal points of Bach but rather segments of musical time that are stationary and have no implication to move ahead; no longer textural constancy as an overlay to harmonic motion but now the freezing of several parameters into miniature eternities. (Kramer 1988: 44)
Kramer has further suggested that a sense of immersion, a still ambiguous term, is achieved in music by minimizing “the distinction between the self and the other” (Kramer 1988: 44), and that this was, in part, made possible precisely by the progressive erosion of linearity in contemporary music (the process detailed in the quote above). In particular, he cites the example of experimental composers in the 1960s who sought not only to diminish the distinction between composers, performers, and listeners, but between the work and the environment in which it is experienced as well (Kramer 1988: 382). Incidentally, his suggestion that dream states, drug use, and certain mental illnesses were formative to such artistic experiments – in so far as suspension of temporality was an aesthetically desirable outcome - resonates acutely with the oneiric nature of TMD’s narrative. The fact that the drama/narrative invokes in the player a state of reminiscence, social uncertainty, and mistrust - and indeed outright hallucination (reliving the memories of a friend) - already somewhat confirms the pliability that the new music idiom, with its requisite vocabularies of immersion and engagement, has with the nature of the narrative in question here. Technology, too, as well as the innovations it has engendered, has been inextricably linked to this reimagining of “time” in modern music. Be it Debussy’s music for the “century of airplanes,” or the enthusiastic appropriation of electronic instruments by modernist composers such as Varèse for (as in just one example) the spatial distribution of sound in specially constructed multimedia performance spaces. Not to mention the contemporary experiments in human-instrument interaction carried out at, among other places, IRCAM.[1] It is an aim of this article to make the case, in part and not otherwise put forward, that such music traditions ought to have a great deal to contribute to any emerging narrative-based mediums that, like TMD, seek to explore the extent to which a player might be immersed. Indeed, this assumption will come to frame, in part, the discussion that is to come of the music itself.[2]
A contemporary development among practitioners of avant-garde and electroacoustic music that is particularly relevant for the discussion to follow is that of soundwalking and the accompanying, though broader, conception of an “acoustic ecology.” Coined by R. Murray Schafer and associated with composers such as Hildegard Westerkamp, a soundwalk is any excursion whose main purpose involves “listening to the environment […] It is exposing our ears to every sound around us no matter where we are” (Schafer 1978: 71). Such a development can be seen to arise out of a tradition which had - simultaneously and perhaps contradictorily - sought to incorporate the extra-musical into an all-embracing compositional practice (say, Varèse) and to problematize the hierarchical modes of mediation associated with established musical practices (typically associated with the concert hall). This could be referring to the Cagean dictum of liberating sound and “erasing the ego of the artist”: relegating much of the actual material and sound-production to improvising musicians, chance procedures, and natural/pre-existent soundscapes (LaBelle 2006: 20). Soundwalking, a capacious term indeed, might simply involve a prearranged (or composed) walking route that factors in the sounds of the environment. It may also be a composition designed to accompany a walk (which may or may not incorporate natural sounds) or a piece of music (electroacoustic or otherwise) that is designed to evoke the peripatetic or ambulatory experience. Such compositions are properly non-linear, often indeterminate, and already repurpose the agency of the listener, not as a passive agent in the listening experience but as an active participant in the creation and unfolding of the composition. Of soundwalking’s radical capacity, John Levack Drever says the following:
The concatenation of locomotion with a state of readiness for attentive unprejudiced listening that protracts beyond the walls of the consecrated sites of dedicated otic practices (that is, concert hall, church, lecture theatre) form the commonality that is soundwalking. (Drever 2009: 163)
Out of this, it is worth drawing particular attention to the concept of acoustic ecology, a way of conceiving sound in terms of a self-sustaining and reciprocal environment. As Schafer puts it, the term “acoustic ecology” suggests that we attempt to hear the acoustic environment around us as a “musical composition and further, that we own responsibility for its composition” (Schafer 1977: 205). It seeks, then, to reimagine space in terms of what we hear rather than what we see (Schafer describes this as the “eye culture”). Enduring and far-reaching in its influence, it gave rise to the idea of a soundscape in which the sounds that we hear in our everyday life are not simply discarded as noise. Instead, such sounds are understood to contain an innate kind of musicality. That this should have some special importance now will become apparent later in the article.
The debates pertaining to non-linearity are not unique to music. A great many of the ideas upon which we have drawn have their analogue in the discussion of (among other things) modernist literature. Espen J. Aarseth, in his discussion of hypertext fiction, has developed the concept of an ergodic literature in which “nontrivial effort is required to allow the reader to traverse the text” (Aarseth 1997: 1). In addition to the textual organization unique to hypertext fiction in particular, an ergodic literature assumes a centering of attention onto the user (of the text) as a “more integrated figure” than might be the case in more typically linear narrative-based mediums. The term ergodic has since found common use in the academic discussion of video games and might prove to be useful here. However, that a text might, as Aarseth seems to suggest, require only trivial forms of engagement, strikes me as a rather troubling assumption. Indeed, any medium - be it of the overtly narrative sort (literature) or the more abstract (say, music) - has its immersive potential conditioned by a complex hermeneutic response unique to, and contingent upon, the faculties and priorities (creative or otherwise) of the individual or communities experiencing it. That this might be the case of traditionally linear mediums is evidenced in the first part of this section. Aarseth elsewhere conceded the dangers of an overinvestment in terms like non-linearity, though: “the linear can flirt with nonlinearity, but the nonlinear cannot lie and pretend to be linear” (Aarseth 2003: 764), and therein lies a curious paradox. A Choose Your Own Adventure book could have the reader flicking back and forth between pages, giving only the most superficial and illusory of choice to its reader, but is this necessarily a richer text than say, Joyce’s Finnegan’s Wake? Joyce’s late masterpiece contains an incoherent overabundance of linearly presented meanings, associations, etc.; the resultant work invites a multiplicity of interpretative responses and rereads.
Non-linearity is by no means a clearly defined or definable term, as I hope to have established above. Moreover, the relationship between non-linearity and the here central principles of “interactivity,” “immersion,” and even “presence” are significant, if difficult to establish; indeed, I have been mostly quite imprecise in my adoption of them so far. This is, however, evidence of a great deal of uncertainty across a range of disciplines which draw freely and creatively from them and offer contrasting, often contradictory, accounts as to what they actually mean, not least in the work on new narrative-based media such as TMD itself. Though I have no intention of giving any concrete assertions on the nature or meaning of these words here, I would like to add some cautionary insights.[3] All classical and modern arts are, in the words of media theorist Lev Manovich, “interactive” by nature. He cites, as examples, “ellipses in literary narration, missing details of objects in visual art, and other representational ‘shortcuts’ which require the user to fill in missing information” (Manovich 2001: 56). Indeed, he goes on to propose a “myth of interactivity,” which assumes of new media, too soon, its capacity to transfer authorship to its user through seemingly superficial processes of interaction. Such myths risk equating “interaction” with the physical act of human-object interface at the expense of psychological interaction.
The psychological processes of filling-in, hypothesis formation, recall, and identification, which are required for us to comprehend any text or image at all, are mistakenly identified with an objectively existing structure of interactive links. (Manovich 2001: 57)
By confusing the internal freedoms involved in the mental reconstruction of the artwork with an externalized, and typically digital, ideal of (an always contingent) choice-making - Manovich offers up Althusser’s concept of “interpellation” - we are asked, in effect, “to mistake the structure of somebody’s else [sic] mind for our own” (Manovich 2001: 61). Of video games, a contrasting, but critically complementary, “immersive fallacy” has been observed. The immersive fallacy contains the widely held but largely unexamined assumption that the “pleasure of a media experience lies in its ability to sensually transport the participant into an illusory, simulated reality” (Salen and Zimmerman 2004: 451),typically, to such an extent that any framing device (say the boundaries on a television) melt away. That this is an incomplete understanding of immersion should seem obvious, and in the context of computer games at least, it is the processes involved in “play” itself that are overlooked. For Katie Salen and Eric Zimmerman, “play is a process of meta-communication, a double-consciousness in which the player is well aware of the artificiality of the play situation." Engagement through play is a far more complex proposition, and the potential it offers this debate may ultimately be more enriching. Tetris is an example of a game (and a particularly enduring one at that) in which immersion is not bound to a “sensory replication of reality” (Salen and Zimmerman 2004: 452). Immersion then should not simply be predicated on a basic suspension of disbelief, and this strikes me as particularly important when trying to carve a place, in projects like TMD, for music. After all, if the goal is the reproduction of some form of immersive reality (or rather, a mixed-reality) – and in so doing the production of a diegetic world unhindered by the non-diegetic – should the musical score, rich in otherworldly associations, be cast aside? Given the abstractions that music typically offers (it is the abstract art par excellence), a broader understanding of the limits of interactivity and immersion ought to be developed, if indeed music is to be a fully contributing and creative force for the future of transmedia and pervasive drama efforts. In the meantime, we should seek to resist casting music as simply the icing on the cake (or at worst, a glorified elevator Muzak!). I hope to reinforce Frances Dyson’s contention, to consider sound or music not “as the precondition for virtuality or cyber space, but to show the way that sound, technology, and culture have combined to create a rhetorical structure through which prior notions of embodiment, materiality, humanity, art, and science are reassembled for deployment in the information age” (Dyson 2009: 7).
[1] Hughes Vinet (Vinet 1999) has given some sense of IRCAM’s priorities. As well as the aforementioned human-instrument interaction, research interests include acoustics, music perception and cognition, synthesis, and computer-aided composition. All of these (as will become clear) bear some relationship to, and could contribute significantly towards, the types of work discussed here.
[2] It is worth noting that a discussion of music is largely absent from the literature about pervasive storytelling. This is likely the case because it is a relatively new medium and theorizers of sound have simply not come around to discussing it yet. Given the capricious and site-specific nature of the format, there has also been little opportunity to preserve what music might have been written. Practice-orientated textbooks intended to provide a survey of past productions and advise on future design of “pervasive games” have little (if any) mention of music or sound (see Montola, Stenros and Waern 2009).
[3] There have been a number of attempts at clarifying the meanings of the terms “immersion” and “presence” within which, it has been noted, there have been inconsistencies (see Slater 2003).
Composing for The Memory Dealer
There is a limit concerning the extent to which I can, and indeed should, talk about my score in isolation. Film music scholarship has gradually come to accept the value of an understanding of the composed-score as one piece in a dialogue with the image and the non-musical sounds it accompanies (and in some cases, is obscured by). This is in contrast to the study of the score, imagined by a composer and abstracted from the film that conditions it: a sort of imagined symphony that could only direct attention away from the immersive intent of any onscreen drama. In film, sound has, unlike the image, no boundary, but is contained in and contingent on the image; for Michel Chion then, there is, properly speaking, “no soundtrack” (Chion 1994: 68). That this insight seems all the more prescient in the context of a transmedia and pervasive experience, where a certain kind of immersion is central to the success of the piece, should seem obvious at this stage. Rick Altman, on the back of Chion’s provocative insight, has emphasized the significance of recognizing the growing importance assumed by the very “intercomponent, intrasoundtrack relationships” of which music provides just one facet (Altman, McGraw and Tatroe 2000: 341). He ultimately proposes a set of mise-en-bande analyses which search for new methods of notation that ought to illustrate the use of sound in film and better understand that it is this “clash of sound elements” that establishes for the film soundtrack “an independent concept and space” in the first place. While it is not possible to do the same in this instance - indeed, the transmedia and pervasive idiom is still too young for any such useful and generalized analytical frameworks - the insights above should not go unincorporated here. While my discussion of the music is conditioned by my experience as composer specifically, it would thus be unwise to discuss my music in purely musico-analytical terms. The discussion that follows will be a practical and explorative one. As with elsewhere in this article, I will endeavor to find support and justification through reference to player interview transcripts.
Conception (and composition)
It mostly gave the whole thing structure. It let you kind of … it strung all the bits together, gave it consistency, like I said, and … yeah, helped it. If there was no music, it wouldn’t have worked; it was totally essential to the experience. (FRA4)
There are a number of things that the TMD score was expected to do from the start. It is important to focus, in part, on a discussion of the goals - both aesthetic and technical - that director Rik Lander and I began with before and during the composition itself. This will better frame some of the critical reflections that will characterize later sections of this article. In so far as this was how it was composed, the score for TMD, it could best be said, emerges out of, and through, the interactions between a basic set of contrasting but interrelated character or mood themes. The effect is ultimately comparable to a branching/layering technique discussed briefly in the context of video game music in the next section. The goal was to produce a kind of master-score composed, in layers, of themes and ostinati that have certain characteristics on their own but can function differently together and in different combinations. A discussion of these themes will follow.[4]
Figure A and AudioObject1: Theme 1 (character Eve).
Theme 1, detailed in figure A above, is the basic melody most recurrent throughout. It is typically associated with the character of Eve and stands generally for any kind of reminiscence in the drama. The basic rhythmic design - consecutive quavers - has an inherent ambiguity, as does its E Mixolydian modal character. That is to say, the written meter is not innately present in the shape of rhythm and contour (despite the presentation above). This ensures that the melody lends itself well to types of modal and rhythmic manipulation albeit via potentially simple means. The next example, figure B below, consists of another melody cast in the E Mixolydian mode. A second voice accompanies it with a classical mix of contrary and similar motion, but with an emphasis on major and minor second intervals: a dissonance allayed by octave displacement and the modal harmonic environment in which the music here operates. Metrically, the melody is symmetrical, and the first and last bar read like near-transposed retrogrades of one another. However, it is thematically asymmetrical, and a downward contour for the first two bars is fragmented in the following two. Over all of this, phrasing and articulation send further conflicting – or rather, overlapping – messages. This melody remains, however, consistently on the beat. Many of these characteristics are comparable to those of theme 1.
Figure B and AudioObject2: Theme 2.
Figure C below is an accompaniment motif with melodic qualities. It was conceived as a kind of multi-accompaniment that could support, if dramatic demands called for it, all three melodies discussed in this section. Likewise, it contains a number of intentionally subtle rhythmic distortions, such as the syncopated E pedal (which likewise reinforces the E Mixolydian character), the syncopations in the opening E-D-C# - which are corrected two bars later from the last beat of the third bar – and the near-repetition with a slight rhythmic variation on the fifth bar.
Figure D and AudioObject4: Theme 3
Theme 3 in figure D above is a final theme with a contrasting E Phrygian character; it, however crudely, signals negative themes and characters. The shared E as a type of tonal center means that, though it is distinct in terms of mood when compared with the previous themes discussed, all materials remain closely related.
The audio example below is a composition consisting of the above themes. As it progresses, the themes are layered (added/removed) and played in a number of different combinations. The effect involves a kind of “layering” or “branching” to tie together interrelated materials in a potentially indeterminate way.
AudioObject5: Layered composition.
A basic composition with a number of simultaneous layers can be dissected and rearranged to produce new compositions of varying lengths; it has the capacity to traverse a range of moods and associations, and this is done via a surprisingly simple process of placement and reordering. The argument I make here is that the basic and static musical materials that have been composed – though complex in their subtle treatment of features like rhythm, modality, shape, and contour – are capable of producing in themselves variations when recombined and re-contextualized. Such different combinations are capable of drawing out from these themes and accompaniments different latent characteristics, moods, and associations. When a score needs to be assembled on very short notice for a drama that is prone to considerable alterations between performances, it is possible with this approach to produce long and branching scores that can still respond, through such basic processes as repetition, to the indeterminate demands of a non-linear narrative.
Figure E and AudioObject6: Transition 1.
AudioObject 6 contains a transitional passage that acts, by way of example, as a connector for major-sounding and minor-sounding materials detailed above. This type of material is useful if a certain kind of linear, and ostensibly filmic, developmental progression is needed. In the example, theme 1 is destabilized by a Phrygian figure, which is itself a variation of that very theme. This evolves into a full blown poly-modality at the very point in which the texture is at its thickest. By the time the stripped down material detailed in figure E has arrived, the music has sidestepped entirely into an E Phrygian musical territory. The material in figure E shares its static character with the other themes and can be repeated where necessary to fit the changing temporal demands of TMD.
In addition to the composed materials detailed above, I accumulated, by way of a contingency, a catalogue of related sounds that could, on short notice, be treated (electronically or otherwise) - that is to say, looped, stretched, etc. - and quickly combined/recombined to produce a bespoke score. These involved live, and typically timbral, samples recorded by student string ensmbles from the University of Nottingham Music Department. Such samples included, for example, a number of slow and evolving “drones” on the E root (the “home” key in the figures above) as well as improvisatory passages based upon the character themes.
AudioObject7.
AudioObject7 contains numerous recordings of drones (recorded by a string orchestra and incorporating a number of slight timbral and tonal inflections, of a progressively more unsettling and dissonant type) supplemented by a separate recording of theme 1 improvised on a cello. Again, such material is simply combined, easily rearranged, but filmically ostentatious in presentation.
[4] It is, I think, important to note that when the project began, and for much of the time during which I had planned for the music to be composed, Rik Lander and I had very little idea about how the score was to be implemented or experienced. This is symptomatic both of the exploratory nature of my involvement with TMD and the director’s conception of how such a drama should operate. There is - it should become clear - a kind of conflict between a site-specific work that is, necessarily, developed from the ground up on each and every new performance, and the need for a coherent and through-composed score, which typically requires a great deal of contemplation and work to produce adequately and effectively. Indeed, it is the question of how to navigate this difficulty that is central to the current discussion.
New approaches to music and immersion
Research into the use of music in video games, a growing body of work, provides arguably the clearest articulation, theoretical and otherwise, of how music can accompany non-linear and “ergodic” new media narratives. Indeed, the video game itself, the most ubiquitous and popular new media platform, has behind it a now strong tradition of musical accompaniment with established techniques and vocabularies distinct from the not unrelated and equivalent practices in film music. Because of the player-centric approach to TMD and its own particular narrative approach, which is entirely indeterminate, locale-specific and/or delivered in headphones as a player explores environments independently, the framework offered by video game scholarship is especially valid here. The most useful insight we can draw from video game musical practice concerns the means through which a score can operate in a fundamentally indeterminate environment, wherein player choice and interaction/agency must necessarily be accompanied by a, at least seemingly, responsive, dynamic or adaptive soundtrack. To this end, four strategies have been observed for the creation of an adaptive music in the context of video games. These are branching, layering, transitions, and generative music (Geelen 2006: 96). By way of a brief summary, branching requires a multi-layered score-track in which choice layers of music may be added or subtracted depending on progress or whereabouts in the game world. The effect is a continual and seamless transition. Layering involves a basic composition onto which additional instrumental layers are added by way of a development in tension and triggered by player interactions and narrative progress. Transitions involve shorter compositions that connect often unrelated and largely repetitive compositions; these would also be triggered by in-game player decisions. Transitions effectively smooth over potentially incongruous changes of music. Finally, generative music – a type of algorithmic composition – would involve, effectively, teaching the computer to write its own score. This could be achieved through the use of, say, Markov models/analyses in which “the type and quality of the output will depend largely on the properties of the corpus and can also be predicted very well in comparison to other procedures such as neural networks or cellular automata” (Nierhaus: 2009: 81). In this sense, generative music requires specialized knowledge and has typically been a feature of “indie” games that are in some sense explicitly about the music.[5]
Video game scholarship has often cited a tripartite model for immersion. A proposed topology of this type incorporates sensory immersion, challenge-based immersion, and imaginative immersion (though these categories are not mutually exclusive).[6] Music operates, it seems, most effectively at the level of the sensory, where it functions primarily to “overpower the sensory information coming from the real world,” allowing the player to become “entirely focused on the game world and its stimuli” (Ermi and Mäyrä 2006: 7). Challenge-based immersion is most unique to the video game genre and the more directly interactive component that typically characterizes it. This is best achieved through an emphasis on motor and mental skills articulated explicitly through the means of a controller-based interface (and indeed challenge, or the illusion of, is not strictly absent from TMD). Finally, there is the imaginary, familiar to all who have enjoyed fiction of any sort. It concerns our capacity to engage and empathize with a fictional world or narratives whether in the act of, say, reading/playing or as you recall the experience while you go about your daily routine. It is interesting to note that audio, if it is indeed to effect negatively a player’s engagement within a fictional world, is most likely to impede precisely the imaginative level of immersion. Sander Huiberts has, in a study of video game players, cited several instances in which audio has negatively influenced “imaginative immersion by diminishing the player’s empathy with the character, setting and story” (Huiberts 2010: 111); this is a thread that will be picked up on later in the article. Gordon Calleja offers, by way of a solution to some of the criticisms of immersion discussed in the previous section, his own “player involvement model” (Calleja 2011: 35). The model incorporates six types of “involvement,” three each belonging to a micro and a macro layer: these are ludic, kinesthetic, and spatial involvement (micro) and affective, narrative, and shared involvement (macro). Involvement, he claims, is a “prerequisite to the experience of higher-order cognitive processes such as presence or immersion in much the same way that attention is a prerequisite of involvement” (Calleja 2011: 35). That these might be applicable to TMD is evidenced by the inclusion of “out-of-game” types of involvement. This is a central focus for the transmedia and pervasive idiom, wherein personal exploration of websites, posters, mobile phones, and magazines outside of game time becomes an intrinsic part of the narrative experience. Such a multifaceted approach to immersion offers, for the musician, a striking array of approaches and mediums through which music could engage, color, and construct a dialogue with the narrative and its participants.
The experienced soundscape of the modern video game is never the same; this contrasts with sound in film that is typically fixed at its point of production (barring instances of re-edits, dubbing, etc.) (Grimshaw 2012: 350). A video game score, be it generated procedurally or assembled from a library of audio samples triggered by in-game decisions, is composed in the moment and achieves its own peculiar sense of immersion precisely through the spontaneity inherent to interaction. The soundtrack of TMD, in so far as it facilitates the interaction of player and narrative, sits uncomfortably between the above poles of video game and film. On one hand, the musical content of the soundtrack is – upon download of the app and barring some incidents in which music is triggered by player movement – largely determined. But the intended dramatic experience as a whole, which involves indeterminism in time and space as well as the inclusion of choice and agency for its players, demands of – and instils into the score – a corresponding non-linearity which, it might be said, renders it qualitatively different from something like film. That this results in a music that is experienced to be indeterminate (when in fact, it might not be) should seem a moot point. Indeed, that truly adaptive music was not strictly implemented here had, ultimately, more to do with budgetary and technical constraints than with intent and design. Ecological modes of listening already imply, anyway, the way individuals will frequently use music to mediate perception of their surroundings; listeners are capable of naturally “aestheticizing” an environment by “soundtracking” it (Herbert 2012). Ruth Herbert cites an example of a research participant doing just this in a car journey while listening to Reich’s Music for 18 Musicians. The objective qualities of a minimalist musical style (chiefly repetition) accord particularly well, as though choreographed, with the movement of the world outside this participant’s window (Herbert 2012). This echoes perhaps my own decision to adopt a minimalist aesthetic for this score; the repetitive structures are not simply a practical expediency. TMD’s score is adaptive in so far as it is an extension of the technologically mediated listening habits that characterize the present day.
Technological constraints, and the influence they exert on the development of music for a narrative-based medium (be it transmedia and pervasive drama or video game), should not go uncommented upon here. Indeed, the question of a hard technological determinism has been a central debate in the burgeoning video game music scholarship (see Collins 2008: 5). This is nothing new, and it has been acknowledged that musicological and historical discussions of film music have typically (and until relatively recently) failed to acknowledge the influence that then-emerging technologies have had on the formation of the film-scoring practices that we take for granted today. James Lastra has, in his technologically-focused history of American cinema, observed that “individual studies of specific media tell us [...] that their technological and cultural forms were by no means historical inevitabilities, but rather the result of complex interactions between technical possibilities, economic incentives, representational norms, and cultural demands” (Lastra 2000: 37). It is important, then, not to assume the integrity of the score in an absolute and abstract way, and the insight which lays bare the very contingency (and contextuality) which conditions the role music can play is indeed a valuable one here. Not least to ensure, at an early stage in this research, a critical and reflective practice, but also because the technological constraints have been particularly noteworthy. Indeed, a drama with the goal of an absolute and bodily/cognitive immersion without even the minimal interfacing required of virtual-reality or a hyper-realistic computer game (as is the case for TMD) can arguably best be characterized by, and in reference to, its system of constraints. The extent to which these constraints exerted an influence over the composition and production of the TMD score then should not go unrecognized.
[5] A good example here is the score for the game Spore (2008), composed by no less a name than Brian Eno. Created with the Pure Data visual programming language, a piece of software known as “The Shuffler” creates a procedurally generated score from a bank of samples.
[6] As noted, a tripartite structure is a common way to do this. This particular one comes from Laura Ermi and Frans Mäyrä (Ermi and Mäyrä 2006).
Composition in practice and on reflection
The section that follows will examine some of the practical outcomes and insights that performances of TMD, with a completed score, provided. I shall attempt to identify and assess where the score succeeded and where it might have faltered to incorporate these observations into a practical framework applicable to future projects of a similar nature.
Beyond providing a dramatic continuity across a changing transmedia narrative, the score proved highly effective at simply reassuring the audience that their mobile device was still operational. Immersion is therefore safeguarded, to some extent, via a filmic convention of a music that is almost always there. More than a reinforcement of the commonplace assumption that music provides an auditory cushion for the “viewer’s sensory background, that area least susceptible to rigorous judgement and most susceptible to affective manipulation” (Gorbman 1987: 12), music here provided a kind of practical and technical support which I had not foreseen at the beginning of the project. In practice, this was an example of where the music excelled, and a number of quotes from the interview transcripts (below) give first hand evidence of this.
That is part of why the music is good, because it shuts out the peripheral stuff that isn’t part of the story. So by taking it out when you actually talk to someone in the story, you can then just plug yourself back in, and you … (FRJ5)
… it was waiting for me to get to the next level. So for me it was like a signal and an indicative point where you finished a chapter, in a way. It is a bit like, you know, audio books where you, you know, the chapter ends, and it is music, and you have to skip to the next one. It felt like it was being used to […] but I kept kind of going back to it because I wasn’t sure when […] because you’re not sure what is going to happen next … (FRK2)
… it helped me to understand the words and characters and things, and sometimes it was not very clear from the headphone, so the music helped me to figure out what the characters were thinking about … (SUH1)
It seems important that the score appeals to a basic and readily understandable shared musical vocabulary. Experimental new media of this variety is no doubt prone to technical faults, and a typically (at this stage) expectant audience may already be waiting for something to go wrong. It is an ever-present threat in a project like this that the hypothetical and fully-immersed pervasive drama player would, if the mobile device stopped working, proceed under the assumption that everything was still functioning as it should. Perhaps paradoxically then, the actual player is in an already – though perhaps necessarily – skeptical state. Music, it seems, has the unique capacity to reassure an audience that a fictional world remains in operation without breaking the narrative rules that lend that reality its substance.
Because the drama was interwoven into, and in some sense derived from, its surroundings – and in part, due to its still experimental nature – it became a requirement that TMD be, to some extent, reconstructed for each and every performance and re-manifestation. This applies also to the score, which was, as I have discussed, effectively recomposed on a number of occasions. This brings me to an important observation that may be, perhaps, unique to this project. That is to say, there was a need for a compositional model to be established, from the outset, capable of providing a score that is both
(1) dynamically responsive – not strictly to the in-world decisions of any individual player or agency – but to the changing, culturally contingent, technically fraught, and often capricious demands inherent in site-specific, multimedia-based narrative environments, and
(2) while being simultaneously filmic (linearly so, in terms of layered flow), functionally adaptive (though probably only delivering an illusory sense of this) and willing to carry the kinds of seemingly universal and yet reassuringly cinematic/semantic codifications which ensure a direct means by which to communicate affective, and occasionally technical, messages to the player(s).
This was, in part, covered in the previous section, and the layering and branching techniques, if applied properly, constitute something of a solution for this problem. In addition to that was the use of audio samples, again also already discussed. In practice, the production was not complete until the night before the performance itself; that this has profound implications for the design of the music should go without saying. It became necessary then to have established a compositional environment befitting a bespoke approach to composition and score design. Having already discussed how the musical materials lent themselves to such an approach, a sequencer software (in this case, Ableton Live) provided the practical backdrop against which this was possible. Figure F shows the working environment, in which a track featuring the voice-over (which was prone to change at short notice) could be supplemented by a drag-and-drop treatment of exported audio samples. Additional mastering and digital manipulation could be applied to the audio within this environment. Though this was an, at times, primitive approach, it facilitated an easy and ultimately economical back-and-forth relationship between the creation of music and other technical aspects involved in the production of the drama.
Figure F: Working environment.
Collaboration means, by definition, a willing surrender of authorial control, or at least, a reconsideration of the term. This is not, however, an observation unique to this project. Indeed, it has always been (to varying extents) the case that composed scores for any comparable narrative medium have been subjected to external forces that might alter the basic construction (and composition) of the material in quite drastic ways.[7] Certainly, though, this is an observation that might, or rather should, have a special resonance here. In fact, it should necessarily follow that a fully reciprocal creative practice (like the one discussed here), co-productive as it is in the evocation of new types of immersion, should likewise be subject to imaginative agencies other than its (in this case composing) author: just as it should influence other parameters of the drama, it must be subject to an influence too. In practice, a lot of the theoretical and conceptual designs of the score were not put into practice in as literal a way as I had hoped (though they might well be latent in the material itself), and ultimately, it was the director who was the final arbiter of how the music was implemented (technical constraints, too, played a limiting factor). This, however, has a great deal to do with the experimental nature of the project as it stands. My approach, which involved the collection of mutually related types of music, actual audio samples both melodic and textural, as well as composed ostinati, was a flexible one, and it was proven possible that where the work fell out of my hands, the director could work with my materials effectively and efficiently. The compositional process necessary for TMD was a fully integrated, and thus, collaborative one.
The following two quotes, drawn from two different interviews with TMD players, detail one way in which the completed score had proven particularly effective.
So I really, what I really liked about it being […] as you’re walking, is that it enables the sounds of the real world to fold into it. And there was someone playing, was it, the guitar, wasn’t it? (FRG1)
… at one stage I thought the saxophone player on the bridge was playing a similar tune. I thought that was amazing! (SUC1)
Both quotes concern the opening act in which the player listens to a pre-recorded audio track through headphones (connected to a mobile phone) and is instructed to walk and explore the immediate area. The music, which adheres to a number of filmic conventions in its support of a narrative-expositional voice-over was, by design, enhanced through the affordance of spontaneous association with sounds generated in the environment. A problem posed in this set-up was that many of these natural sounds are already outside of the headphones and not in the design of the piece itself, though it is a quality of the music – modal, broadly ambiguous in a number of its parameters, unobtrusive, and economical – that it could quite easily interact with the acoustic surroundings.
This type of serendipity was not unique to the score; in fact, the script itself – which called upon the player to enter a state of reminiscence – was able to accommodate a similar effect. A particularly nice example of this is illustrated by the following player quote:
And there was quite a nice coincidence, which was when you were asked to remember your first memories and a small child ran in to me, and there was actually something quite nice about that … (SAB2)
Serendipity is not simply coincidence, nor would it be realistic to suggest that it was possible, in a meaningful way, to create these kinds of opportunities in score or script. It assumes the capacity of the player to be complicit in the creation of their soundtrack, even when, as is the case, it is supplied to them (via headphones) in a linear way. William Gaver has proposed one of two types of listening, derived from an acoustic-ecological understanding of sound. Everyday listening proposes an “omni-directional, semi-distracted, adaptive-interactive listening that focuses on immediate information-processing” (Gaver 1994: 426); the other is a focused analytical listening. The implication here is that an acoustic ecology assumes of the listening subject a kind of creativity. This first act is, then, a kind of aforementioned soundwalk, and it is the semi-spontaneous acoustic ecology of the act (part composed, part in the environment) that becomes the vehicle through which a creative player can encounter, and incorporate into their experience, re-contextualized sounds from a natural environment. When a soundwalk is brought into a narrative framework, as it is here, the potential for coincidence – elevated instead to a kind of serendipity – coalesces with the subject’s agency as a substitute for the dynamic and adaptive soundtracks envisioned earlier in this article.
A similar effect is experienced by anyone listening to music through headphones (for whatever personal reasons). This is consistent with the stated transmedia objective of creating immersion through a real-world-inspired engagement with players and their everyday technical preoccupations and experiences. It is also evidence of the score’s capacity (as a kind of soundwalk) to engage in a commentary on the way digital and mobile media interact with and create narratives for how people explore and experience spaces. The filmic experience, then, comes – already entangled – with the everyday experience of listening to music solitarily and for pleasure and the very ecologies of sound that characterize the contemporary imagination.[8]
[7] Irwin Bazelon notes, in rather dismissive terms (a strong reflection of composerly/authorial bias) that the music “may be cut to pieces, edited and reedited, with different segments spliced together, cues used out of context from their original conception, volumes lowered indiscriminately, balances destroyed or severely altered, entrances and exits changed – everything, in truth, to make a composer hearing the music in conjunction with the film for the first time wonder what happened to the notes he wrote” (Bazelon 1975: 53).
[8] See Jean-Paul Thibaud’s thoughts about the way in which a private (Walkman-based) listening experience is inextricably linked to the individual’s experience of the urban landscape (Thibaud 2003), or Michael Bull’s investigation into how headphones offer listeners the opportunity to create an “auditory bubble” through which they can maintain their sense of subjectivity as they “inhabit” public spaces (Bull 2005: 334).
Choice and space (Good/Bad Eve)
As an extension to a discussion concerned primarily with immersion and engagement (and how music might advance these aesthetically desirable outcomes), I shall for a moment consider the score as a vehicle of choice and influence. That music might have the capacity to encourage certain types of behavior should not be a revelation here. Classical (that is, Pavlovian) conditioning techniques have long been employed in marketing and advertising to make people buy things (to varying degrees of success) and it should be easy to see how music might have been involved in this process (Gorn 1982: 94). The premise is a simple one: set a product against the backdrop of some overtly pleasing music (this might involve major modes and/or positive cultural allusions), and when the customer encounters this product in, say, a shop, these positive associations are invoked.[9] An influence is thus exerted on an individual consumer’s decision making. This is, however, a limited and artistically unappealing type of coercion.
Brandon LaBelle, in a discussion of Muzak and the architecture of sound in shopping malls, has noted that music can act as a form of mental conditioning. Such sonic experiences, like those of elevator music or “moodsong” in film, he says, resituate
our relation to figure and ground, foreground and background by operating on the peripheries of perception, and through conditioning the atmosphere of place, granting the design of space a dynamic psychoacoustic supplement. (LaBelle 2010: 173-174)
In the shopping mall environment, the purpose-built music draws attention to the “effective capabilities” of its other architectural designs and concerns. A different kind of musical non-linearity – built instead upon distracted modes of listening – plays its part in creating the type of space in which customers might relax and direct their attentions in certain, ostensibly profit-making, directions.[10] This potential for a music that refigures a listener’s relation to space, thus reconditioning their choice-making contexts, need not necessarily be a one-way system, concerned purely, say, with commercial gain. What might, for example, be the benefit of these observations for the purpose of telling a story?
To this end, Rik Lander and I incorporated a controlled experiment into two days of the three-day performance run of TMD. On the second and third days we sought, through subtle means of musical suggestion, to attach positive and negative associations (respectively) to Eve, the narrative’s protagonist. It was our hope that this might bear out the contention that music could influence a player’s choice in a concrete way, when during the final moments of the piece a group of participants is given a decision that involves betraying the Eve character or exercising trust in her (and risk getting arrested). Two near-identical scores were provided, both befitting the conditional and bespoke nature of the score as discussed in the previous sections. By altering the mood of the recurrent Eve theme – in this case changing the musical mode from E Mixolydian to E Phrygian (by dropping the accidentals) – two structurally similar but highly contrasting and affective moods were conveyed.
AudioObject8: Eve theme with a minor mode.
These contrasting scores were used at precisely the moments when a distracted listening was likely to take place: chiefly as non-diegetic accompaniment while the players waited and were briefed before beginning the first act and again at the end during the ultimate decision regarding the player and Eve’s fate. The good and bad score variants were also incorporated into the third act as part of a set of interactive installations as well as in the lobby, while players were primed before embarking on the first act. One instance in which said good and bad score variants were used across different performances was “the Car Scene,” wherein players were required to search a police car for clues and were subjected to highly suggestive audio and visual characters and narrative exposition. The music accompanied a video installation and a car radio extract and, in addition to its non-diegetic role, was hereby fulfilling diegetic and meta-diegetic (that is to say, non-diegetic headphone music heard coming out of a diegetic source) functions.
Our findings for this particular experiment have been largely inconclusive, and an examination of player decisions shows no noteworthy trend across different days with different soundtracks. Indeed, the experiment was limited by factors such as budget, technology, and the limited opportunities within TMD into which two contrasting scores could be worked. It might be worth noting however that when players sought to justify their final decision, it tended to have more to do with detached political and/or objective reasoning than it did with gut-feeling and emotional attachment to the Eve character.[11] Responses from those who stayed cited “the greater good,” (SUC6) and were clearly motivated by what they thought might be “correct” according to the moral tone of the story; in contrast, one player who made the opposite decision cited self-preservation and did not “want to go to jail […] if you hadn’t stopped me, I would have run.” (SAA1) Future work may then wish to explore the extent to which music might mitigate this tendency to reflect on, and thus respond to, the story – as seems to have happened here – as a rational observer. Protagonists in films are known to make rash decisions that the sensible viewer might not always appreciate, and perhaps this could be an indicator of success were participants to act more according to impulse rather than with rational decision. Might music be a channel through which to facilitate this?
When asked whether they thought music had swayed their decision-making, subjects of a market-orientated experiment did not seem to recognize consciously the influence music had on their decisions. The results, however, strongly suggested it had (see Gorn 1982). Gerald J. Gorn concludes from this that an “audience may be largely comprised of uninvolved potential consumers rather than cognitively active problem solvers” (Gorn 1982: 100). Indeed, were subjects to “know” that their choice was being manipulated, the experiment might already have failed. The direct influence music could exert may well be difficult to pin down in practical terms, then, but that says nothing about its potential. It is already the condition of the non-linear and sound-ecological paradigm that the ability of music to interact with listener and space is not limited by the traditional categories of listening on which many continue to rely. That music, or the soundscape more generally, is a factor in a subject’s agency is beyond question, but it is a precondition that this influence, experienced holistically and applied artificially for the purpose of conveying a story, should be difficult to quantify: its success being inversely proportional to our ability to recognize it. The implications of this for music and its operation within emerging narrative mediums will likely be the subject of future work. Our experiments here serve only to underscore the need for further inquiry, necessarily of a creative and practical kind.
[9] That this implies an overt simplification of how music embodies (and thus communicates) meaning should be obvious (see Bode 2006 for a critique of this and some arguments as to why a socially construed and semiotic reading of musical meaning can be better employed in a consumer context). Nevertheless, few would doubt music’s efficacy in a marketing context.
[10] Note Barry Truax’s heralding of “the end of the Fourier era,” which is characterized by a “moving away from linear acoustic models, separable and independent parameters of aural perception, and stimulus-response behavioral models towards a recognition of the role of nonlinearity, multi-dimension percepts, and information processing.” (Truax 1994: 178) This observation is as true of postmodern and commercial musical environments as it is of then emerging electroacoustic art music.
[11] One player identified quite simply that it was “curiosity” (FRC4) that prompted his final decision; this certainly implies some kind of engagement (an interest in the story), but perhaps not the desired kind of – ostensibly narrative – engagement.
Conclusion: at the limits of verisimilitude
As noted earlier, music is entirely capable of impeding immersion.[12] Huiberts, via his survey of video game players, has suggested a number of ways in which players are commonly drawn out of their gaming experience as a consequence of the soundtrack. These include the overt and obvious use of sound as a means of suggestion and guidance, i.e. the player becomes overly aware of a mechanical change in the soundtrack as triggered by their decisions and in-game progress (see Huiberts 2010: 106). Repetitive music which becomes obtrusive, especially when overtly happy, fast, or “busy,” is another obstacle, as is the inclusion of an already extant and popular music; this creates the risk of contaminating the experience with a whole slew of cultural connotations and genre expectations not intended by the game designers (granted, this might be a desirable outcome in some circumstances). The following quotations, drawn from interviews with TMD players, are evidence of instances of the music not working to aid immersion that seem to correspond to two of the examples discussed above.
I got a bit fed up that it, when it had a long wait, and it was continuous music, then I put the headphones out, because it was a long, when, yeah. So I think for me the longest wait was in the hotel lobby, so I, I, after I realized it was just gonna loop forever I, you know, I took them out then and … (FRC4)
Um, the difficulty sometimes was when people were talking to you, some of the actors were talking, and you had the music going on, it wasn’t clear whether there was gonna be some event on the soundtrack triggered, so I had to keep listening to this and listening to them, so … (FRD1)
The first of the above quotations details a moment in which repetition in the soundtrack, a technique employed to deal with the indeterminate waiting time between set-pieces within the drama, became a distraction and thus drew attention to itself, motivating the players to remove their earphones and step outside of the experience.[13] In the second quotation, the didactic aspect of the score, which was effectively established early in the drama, seems to have sent conflicting messages to a player. The player had come to infer that they were to receive direction from within the earphones, but so long as the music continued, they were left unsure as to whether they should remove them and engage with aspects of the drama situated outside their headphones.
The potential for music to hinder kinds of immersion is, for the composer or musicologist already immersed in a world of (often quite esoteric) study and commentary about the expressive potential of music, perhaps hard to grasp. Indeed, involvement in this project strongly suggested to me that transmedia and new media artists – concerned primarily with creating a pervasive realism which urges the players to feel truly present within new types of narrative diegesis – might be reluctant to afford music a central role for fear of a perceived inherent unrealism. A sobering reflection on the role of music in, say, film, might be in order; of this, Claudia Gorbman has the following to say.
nondiegetic shot or sound is the exception, not the rule – except in the case of music. Therefore: why music, in the tightly consolidated ‘realistic’ world of the sound film? It gives mood, pacing, emotion, yes – but why is it permitted into the narrative’s regime at all? For one thing, it has history on its side. Music has gone hand in hand with dramatic representation ever since the ancient Greek theater, and no doubt before, in ritual forms. (Gorbman 1987: 4)
Typically, verisimilitude refers to the relative success, for example, of film at creating an “immersive, engaging fictional world of hyper-realistic proportions both in terms of image and sound but also of intensity of emotion and experience” (Droumeva 2011: 140). That music is so effective an accompaniment to this might seem strange, and as Gorbman notes, it is (at least partially) an accident of history (and in the case of early and silent film, the result of a technical constraint) that music became tied to the genre at all. As we noted earlier, the development of the film score, once already established, would continue to develop as the result of a complex historical dialogue with any number of the technological and cultural contexts which condition and are conditioned by it. So too should be the case with immersive and/or “ergodic” mediums, and Milena Droumeva has noted – regarding this question of verisimilitude – that while sound in game has developed to conform to a certain sense of an external reality, it has simultaneously had to construct “a sense of reality particular to games” (Droumeva 2011: 140). What it means to be “realistic” with sound, and especially music, is never such a clear-cut issue. At an early stage in the development of musically-accompanied transmedia and pervasive drama it would be unwise to limit music for fear of what immersion it might hinder, especially when that very immersion, an openly ambiguous term, is a contingency afforded by the interplay of its constitutive and contributory parts over time (and this includes its score). That music can be a co-productive agent in developing new kinds of narrative immersion is evidenced, I hope to have shown, by the historical development of film and video game music. Likewise, such insights can only be taken further through creative and exploratory scores for new narrative strategies like those explored in and through The Memory Dealer.
[12] There is a curious lack of research that supports this, though I feel that it ought to stand to reason that anything can, theoretically, “impede” immersion.
[13] It might be added that the long wait time was likely the result of a technical/scheduling error, of which a number occurred on the days of the performance. The music is not strictly and solely responsible for this break in immersion.
References
Aarseth, Espen J. (1997). Cybertext: Perspectives on Ergodic Literature. Baltimore: The Johns Hopkins University Press.
Aarseth, Espen J. (2003). “Nonlinearity and Literary Theory.” In Noah Wardrip-Fruin and Nick Montfort (eds.), The New Media Reader (pp. 761-780). Cambridge, MA: MIT Press.
Altman, Rick, McGraw Jones and Sonia Tatroe (2000). “Inventing the Cinema Soundtrack: Hollywood’s Multiplane Sound System.” In James Buhler, Caryl Flinn and David Neumeyer (eds.), Music and Cinema (pp. 339-359). London: Wesleyan University Press.
Bazelon, Irwin (1975). Knowing the Score: Notes on Film Music. New York: Arco.
Bode, Matthias (2006). “‘Now That’s What I Call Music!’: An Interpretive Approach to Music in Advertising.” Advances in Consumer Research 33/1: 580-585.
Bull, Michael (2005). “No Dead Air! The iPod and the Culture of Mobile Listening.” Leisure Studies 24/4: 343-355.
Calleja, Gordon (2011). In-Game: From Immersion to Incorporation. Cambridge, MA: MIT Press.
Chion, Michel (1994). Audio-Vision: Sound on Screen (trans. Claudia Gorbman). New York: Columbia University Press.
Collins, Karen (2008). Game Sound. Cambridge, MA: MIT Press.
Corn, Gerald J. (1982). “The Effects of Music in Advertising on Choice Behaviour: A Classical Conditioning Approach.” Journal of Marketing 46/1: 94-101.
Cross, Jonathan (1998). The Stravinsky Legacy. Cambridge: Cambridge University Press.
Drever, John L. (2009). “Soundwalking: Aural Excursions into the Everyday.” In James Saunders (ed.), Ashgate Research Companion to Experimental Music (pp. 163-192). Farnham: Ashgate.
Droumeva, Milena (2011). “An Acoustic Communication Framework for Game Sound: Fidelity, Verisimilitude, Ecology.” In Mark Grimshaw (ed.), Game Sound Technology and Player Interaction: Concepts and Developments (pp. 131-152). New York: Information Science Reference.
Dyson, Frances (2009). Sounding New Media: Immersion and Embodiment in the Arts and Culture. Berkeley: University of California Press.
Ermi, Laura and Frans Mäyrä (2005). “Fundamental Components of the Gameplay Experience: Analysing Immersion.” Suzanne de Castell and Jennifer Jenson (eds.)Proceedings of Changing Views: Worlds in Play: Digital Games Research Association's Second International Conference, Vancouver: DiGra and Simon Fraser University.
Gaver, William W. (1994). “Using and Creating Auditory Icons.” In Gregory Kramer (ed.), Auditory Display: Signification, Audification, and Auditory Interfaces (pp. 417-446). Reading, MA: Addison-Wesley.
Geelen, Tim van (2008). “Realising groundbreaking adaptive music.” In Karen Collins (ed.), From Pac-Man to Pop Music: Interactive Audio in Games and New Media (pp. 93-102). Aldershot: Ashgate.
Gorbman, Claudia (1987). Unheard Melodies: Narrative Film Music. London: Indiana University Press.
Gorn, Gerald J. (1982). “The Effects of Music in Advertising on Choice Behavior: A Classical Conditioning Approach.” Journal of Marketing 46/1: 94-101.
Grimshaw, Mark (2012). “Sound and Player Immersion in Digital Games.” In Trevor Pinch and Karin Bijsterveld (eds.), The Oxford Handbook of Sound Studies (pp. 347-366). Oxford: Oxford University Press.
Herbert, Ruth (2012). “Modes of Music Listening and Modes of Subjectivity in Everyday Life.” Journal of Sonic Studies 2/1.
Huiberts, Sander (2010). “Captivating Sound: The Role of Audio for Immersion in Computer Games” (Doctoral Dissertation). Utrecht: Utrecht School of Arts.
Kramer, Jonathan D. (2008). The Time of Music: New Meanings, New Temporalities, New Listening Strategies. New York: Schirmer Books.
LaBelle, Brandon (2010). Acoustic Territories: Sound Culture and Everyday Life. London: Continuum.
LaBelle, Brandon (2006). Background Noise: Perspectives on Sound Art. London: Continuum.
Lastra, James (2000). Sound Technology and the American Cinema: Perception, Representation, Modernity. New York: Columbia University Press.
Manovich, Lev (2001). The Language of New Media. Cambridge, MA: MIT Press.
Montola, Markus, Jaakko Stenros and Annika Waern (2009). Pervasive Games: Theory and Design. Burlington, MA: Morgan Kaufmann Publishers.
Nierhaus, Gerhard (2009). Algorithmic Composition: Paradigms of Automated Music Generation. Vienna: Springer-Verlag.
Salen, Katie and Eric Zimmerman (2004). Rules of Play: Game Design Fundamentals. Cambridge, MA: MIT Press.
Schafer, R. Murray (1977). The Tuning of the World. New York: Knopf.
Schafer, R. Murray (1978). The Vancouver Soundscape. Vancouver: ARC Publications.
Slater, Mel (2003). “A note on presence terminology.” Presence Connect 3/3.
Thibaud, Jean-Paul (2003). The Sonic Composition of the City. In Michael Bull and Les Back (eds.), The Auditory Culture Reader (pp. 329-341). Amsterdam: Berg Publishers.
Truax, Barry (1994). “The Inner and Outer Complexity of Music.” Perspectives of New Music 32/1: 176-193.
Vinet, Hugues (1999). “Recent Research and Development at IRCAM.” Computer Music Journal 23/3: 9-17.