Introduction and Research Questions
How does human movement drive the phenomenology of space? Can interactive video modulate specific aspects of this phenomenology? This paper presents examples of ongoing artistic research that explores movement perception and its influence on our experience of our surroundings.
My artistic research is built on the theory that movements, responding to visual stimuli and motor planning, are processed in parallel and functionally distributed pathways[1]. Current neuroscientific research suggests that action comprehension consists of a vast library of representations – motor and sensorimotor, visual, and semantic - that connect to build our full understanding of movement[2]. In other words, motor imagery, visual imagery and motor production are discrete neurological functions that contribute to diverse facets of our understanding of movement.
In addition, over the past twenty years, research on processing of visual perception of motion and motion performance have prompted speculation about a new class of neurons, “mirror neurons,” which suggest that “action understanding” is facilitated by internal simulation enlisting both motor and sensory systems in the brain[3]. This research supports the idea that the neural description of an action includes motor and visual components- in other words both a participant’s view and an observer’s view[4]. Further, there is mounting evidence for the theory that motor-imagery is not only function-specific but also body-specific[5]. “According to the body-specificity hypothesis, people who interact with their physical environments in systematically different ways should form correspondingly different mental representations.”[6] So a commuter who repeatedly performs at a location rather than simply passing through that location, may gain a unique impression / understanding of it over time.
Based on this scientific framework, I hypothesize that interactive video works eliciting body movements, alter our experience of space. Examining how this happens can help us better understand some of the basic contradictions in motion perception.
- Take a kick. How do we “perceive” it as a concrete, highly physical, action, but “conceive” of it by re-constructing a complex multi-modal lattice of representations and memories in our mind – that might link the sound, the feeling, the look and the impact or purpose of that kick?
- Take a breath. Is this movement instantaneous or enduring; past or present; continual or discrete; individual or collective; something we see, do or know?
- Take a fall. Can different modalities (motor, visual, semantic) trigger simultaneous but different comprehensions of the same action?
The experience of movement has a fractal-like capacity to scale, to be one thing and its opposite simultaneously (now and just now, an internal sensation and an external form, a unique instantiation and a universal language). It is a physical manifestation of the human condition that shifts continually from present to past – the physical shell for our “ephemeral” state of being alive. Movement also conveys a purity of expression. How we move is both idiosyncratic and distinct, like a portrait. Movement physically expresses our unique qualities of energy, marking how a soul “issues forth” out of its body, in an irrefutably truthful form of self-expression.
Performance, and dance in particular, is a highly traditional means for framing the body in action. The stage acts as a physical framework to convey narrative, social, political, physical and emotional drama. The viewer, by remaining still and separate from the moving artwork, can readily identify the artistic frame and vantage point that is prescribed.
[1] Andrew Schwartz. “Distributed Motor Processing in Cerebral Cortex,” Current Opinion in Neurobiology, 4 (1994): 840-846. Prut, Perlmutter & Fetz. “Distributed processing in the motor system: spinal cord perspective,” Progress in Brain Research 130 (2001): 267-278. Andersen, Snyder, Bradley, Xing. “Multimodal representation of space in the posterior parietal cortex and its use in planning movements,” Annual Review of Neuroscience 20 (February 1997): 303-330. Marsel Mesalum. “From Sensation to Cognition,” Brain 121 (1998): 1013–1052. De Lange, Hagoort, and Toni. “Neural topography and content of movement representations,” Journal of Cognitive Neuroscience 17 (2005): 97–112.
[2] Op. cit., Damasio et al.,168.
[3] Giacomo Rizzolatti and Laila Craighero, “Mirror neuron: a neurological approach to empathy,” Neurobiology of Human Values (2005):108-109.
[4] Ibid., 109.
[5] Roel Willems, Peter Hagoort and Daniel Casasanto. “Body-specific motor imagery of hand actions: neural evidence from right- and left-handers,” Association for Psychological Sciences (2009): 1-8.
[6] Daniel Casasanto. “Embodiment of abstract concepts: good and bad in right and left-handers,” Journal of Experimental Psychology General 138 (2009): 351–367.
Despite idealized notions of the commons as a place of meeting and sharing, public space has in fact, always been a location of discord and conflict, ‘characterized by a history of violence just as frequently as it has by more serene activities and civil conduct’ [1]
‘Furthermore, we need to understand how the growing infrastructure of digital displays influences the perception of our public spaces' visual sphere’ [2]
‘What influence does the medium of video have on our experience of architecture?’ [3]
[1] Antoine Picon, ‘Architecture and Public Space Between Reassurance and Threat’, in Journal of Architectural Education, 61:3 2008), 6–12. Article first published online: 10 Jan 2008.
[2] Mirjam Struppek, 'Urban Screens - The Urbane Potential of Public Screens for Interaction', intelligent agent, vol. 6 no. 2, Special Issue: Paper presented at the ISEA2006 Symposium (2006), p. 110.
[3] Catrien Schreuder, Pixels and Places Video Art in Public Space, (Rotterdam: NAi publishers, 2010).
The i:move series 2 (2004)
by Nell Breyer, Jonathan Bachrach, Goran Bogdanovski and Dejan Srhoj
An interactive installation and performance at Dance Theater Workshop Gallery
Made possible (or Made possible in part) by the Grants Program of the Council for the Arts at MIT, The Trust For Mutual Understanding Suitcase Fund, Dance Theater Workshop.
Monday: Change
Stand still, you disappear. Move, you see the difference between where you are now and where you were just now. With stillness, you disappear from the projected field. By moving, you generate the form your changes make in space over time. Movement reveals the architecture behind it. The mover masks, defines and reveals what is underneath. A slow decay of the first image/location over three seconds, creates the impression of a continual transition between points in space.
Tuesday: Volume
Moving, you push air continuously. If the displaced space were clay, this is its shape.
If you imagine that movement continually pushes the air around you, and that air was clay, this visualization reveals the negative space or shape of the displaced air you have moved through. A slow decay over three seconds creates the impression of a continual transition and smooth trail off of the volumetric shape. You see an enhanced, immediate history trail of your movements.
Wednesday: Time
We construct movement through our memory, recalling physical sequences over time. How do we determine the beginning and end of motion? Wednesday we presented a visualization of time, exploring how we re-construct movement through short-term memory. We created a small memory cache of video frames. The cache looped, so that one starting point was repeated three times. The video played over itself four times, continually moving with the viewer and catching up with what we might call ‘real time’. Real time ran in black and white.
Thursday: Meaning
Symbols can notate and delineate movement meaning. Actions demarcate intent. Both signs and actions can represent narrative and abstraction simultaneously.
Thursday varied movement notations underscored the semantic content of an action. As viewers moved by the wall, their movements would act like an erasure, wiping off the blank wall, revealing an under layer of words or signs. We included dancers' explanations for why they move, hand gestures, laban notation and common sign postings signaling well known signals (stop, left, ok etc).
Friday: Contour
The very farthest edges of your movements are drawn in real time. The span of your periphery is continuously tracked, traced, dissolved and reprised.
Friday we visualized movement contours. A heavy-weight line was drawn in real time as you moved, drawing the extreme edges and periphery of your movement through space. The outline would follow your location and shape in real time and rely on the gradual decay to create a continuous band around the breadth of your activity.
Interactive video in public space physically draws out many provocative time-space metaphors. With this type of dynamic visual-kinetic feedback, for example:
- Time can collapse
- Objectivity can reverse
- Boundaries can bend, and dissolve
- Action sees
- Objects, actions and space can fuse and synthesize
- Fragments can constitute the whole, and the whole can be perceived through reorganizing fragments
How does human movement drive the phenomenology of space? Can interactive video modulate specific aspects of this phenomenology?
Interactive video taps the ability of the corporeal body to anchor our attention and sensation. This exposition offers observation-based terms to delineate the impact of interactive video on our experience of public architectural spaces.
We perceive a jump as a concrete, highly physical, action, but we conceive of a jump by re-constructing a complex set of representations in our mind. I suggest that interactions which produce simultaneous visual-motor feedback effect the way we understand our own movements and their impact on our surroundings.
The installation transformed movement in real time, using a unique system architecture that combined movement-centric parameters to extract the perceptual features of motion from video. Human traffic created and continually reshaped the form and flow of the display.
The installation at DTW transformed a lobby area into a mirror through which the viewer negotiated between two views of themself ; visually and physically. Because both spectating and performing were happening inside and outside the foyer, the wall between the theater and the street became porous. Viewers began to manipulate the look and feel of their surroundings and their own image within this.
Interactive video can create an experience of transitional spaces that responds specifically, immediately, and viscerally to the individual. This offers a potentially intimate and relational entry point into our media-saturated urban landscape.
The ephemeral, responsive nature of our cities and their navigation systems, gives particular fluency to interactive video in public spaces.
The i:move series 2 at Dance Theater Workshop is one of a series of public interactive installations I have helped build, exploring motion perception and its impact on our understanding of public liminal spaces. The sites for these work s are passage zones: bridges, walkways, lobby areas interlinking destination locations, but not destinations themselves.
An important precursor to the project at DTW, was the i:move series 1, created by myself and scientist, Jonathan Bachrach, for the Media Lab Plaza at Massachusetts Institute of Technology. We presented this work in 2003, as a large-scale, approximately forty by fifty feet interactive video installation open to the general public. It ran every night for two weeks as a layered, live and pre-processed scripted/improvisational public street performance.
Many of the concepts and system design features initiated preliminarily at MIT, were developed to become more robust, specific and differentiated in the architectural context and movement-savvy audience provided by Dance Theater Workshop in New York.