Movement around the piano in a full circle. The movement is both kept in its original form and cut up into microsamples of the movement, showing different positions along the line of motion. The video on the right side, shows the full motion in combination with the appearing mircosampled positions, like frozen instants of the movement. This may connect loosely to the photo studies of Edward Muybridge, though the images are presented here in a more interactive way: the positions are played out.
The example above illustrates the relation between individual samples, their mapping and how they come together in a performed piece.
The two octaves contain a register of samples from a specific playing technique mapped onto the keys. In reality the 24 black and white keys of the two octaves should represent 24 samples in contrast to the 14 represented here. Still it gives an impression of the individual samples and the subsequent video keyboard version. This particular piece, entitled Force, is part of the album Studies For Video Keyboard which makes up one of the published works of the project.
It should also be said, that his particular piece is a rare example of a full length track where speed/pitch of the video samples is not manipulated.
The piece Rerendered is an important backdrop in how it frames and visualises different approaches to piano playing. We not only hear but see the body's interaction with the instrument.
Rerendered also contains some of the "silent" actions (a gliss of fingernails across the keys, a foot pressing down a pedal without any note) which I make use of. Even if some of these techniques were already part of my vocabulary before watching the piece the first time, Rerendered expanded my understanding of the communicative potentials of these "non-playing" actions, simply by how the techniques were framed and visualized.
The stage setting is also apparently linked in how the pianist is placed center stage with her back to the audience, sharing the audience's direction of view toward the screen (and score). Pianist as both player and, so it seems, spectator.
For more contextual considerations on the link between Rerendered and my (frame)work, please visit the Frameworks section in Practice Grounds.
Another important aspect of the mapping process is to translate the topographic approach centred around my acoustic piano practice to the digital context of the video keyboard which is devoid of the resonant piano body and its mechanics. One thing they do share, is the same keyboard architecture: 88 black and white keys. However, the digital keyboard being eminently virtual, in as much as its keys trigger content which is digitally assigned, opens the possibility to map dimensions differently.
Rather than seeing the horizontal axis of the acoustic piano keys as an outlining of a chromatic pitch scale in even steps, the keyboard is now divided into sections or regions containing different types of material. Each region of the keyboard, say an octave, is designated as a territory containing a certain register of material, say, a specific camera angle, playing technique, sampling approach etc.
The limits of an octave, which in relation to the full playing range of the acoustic piano is only a partial sonic register of the instrument, now contained a whole terrain of images and sounds, also keeping in mind that one key could trigger a full pianistic event. This opened for both section-specific explorations and a playing across sections, interweaving camera angles, images, colors, lights, sounds, rhythms, movements.
The keyboard plane now becomes an intersection of possible playing directions, where the relation between the global keyboard and local key content is radically changed, calling for other orientations of play.
In the Run Time Error pieces (see version above) he moves along a planned route where the architectonic surfaces along with various materials, both placed and misplaced, are "sonified" by his contact.
As such, the routes are like sonic lanes or tracks, which are activated by movement and touch. What we hear and see is the player in motion, and the sounds produced by that motion. Something that relates to stickworks and my interest in the resonating contact points between materials and bodily motion. In Run Time Error, we not only see and hear the movement through a "sonic route" but we also experience a live treatment of those sonic motions. There is a sense of parallelity, or simultaneity: The screen is split in two with basically a double up of the same video. And furthermore, the performer "on screen" is played by a performer "on stage", who controls the two videos with a joystick designated for each one. The joystick is a controller that can loop, rewind, speed up and slow down the videos independently, manipulating both pitch and image and creating a kind of two-part polyphony of the two identical videos. One being a reflection of the other and vice versa.
The features of looping and speed/pitch control are also essential features of the video keyboard. The touch sensitivity of the keys allows for a variation in the speed/pitch and the looping function is both designated to the keys (a key pressed down sustains the video by simply looping it until the key is released) and to one of the pedal functions, which can loop a given duration of a video sample when pressed down.
IV. CONTEXT
If a larger context is laid out in the section on Networks, some of the concepts and performative approaches of the video keyboard point to the audio-visual work of composer Simon Steen-Andersen. Three titles are of certain interest: Rerendered (2003), Piano Concerto (2014) and his growing collection of pieces entitled Run Time Error (2009).
I. BACKGROUND AND MAKING
A synaesthetic turn
It is important to underline that the video keyboard was neither a concept nor a concrete intention when embarking on this research project. As illustrated in PRACTICE grounds, the acoustic piano was the instrumental frame within which the research practice took place.
However, when looking back on the documentation process as it developed from audio recording to lo-fi single-camera footage and onwards to multi-camera sessions in a more professional setting, every step seems to have prehended the video keyboard.
One vital turning point is marked by the shift from audio to video documentation. Turning on the camera, all of a sudden, elucidated my practice, giving way for another kind of listening, a visual listening, where the image seemed to amplify the playing situation: audible gestures becoming visible movements, sonifying the corporeal.
Sound was only one of more possible entrances into the work, and by adding images, the project moved towards a synaesthetic territory, which in turn suggested a an interpretation of multilayeredness where perceptions of sound, image and bodily positioning and movement, converged in a performative terrain different from conventional solo pianistic settings.
A performative flip
Still in time before any notion, at least to my attention, of the video keyboard, we did a multi-camera session in which I tried to cover the practice grounds, spanning from the ritualized routines to open-ended improvisations, going through a repertoire of positions, movements, techniques and playing approaches (stickworks, footworks, fieldworks, fireworks).
The approach was quite different from how I would normally enter a recording session. Instead of leaving the practice behind in pursuit of a work of art, I was curious to see what would happen if I went into the session with a kind of practice-ritualistic approach. Preparatory and intermissionary activities, such as entering and leaving the room, sitting down at the piano, opening and closing the lid, resting quietly before play, were also filmed.
The purpose was two-fold. On the one hand, it was to investigate the expressive potential of the seemingly simple actions and exercises, seeing also if the different camera angles would communicate these relations differently. On the other hand, I was curious to see how the longer stretches of improvisation would communicate as a solo performance on film.
When I received the session files, I experienced both excitement and disappointment. It was eye-catching to see the clear-cut angles, the precision of framing and sharpness of focus. The framings, especially the overhead view, seemed to illustrate what I had envisioned: the choreography of play, its relations of movements and velocities within and across the topographic field of play, somehow poeticizing the encounter between player and piano. And yet, the material still carried with it a sense of documentation, an observation of the activity of play, lacking both communicative direction and expressive force. Or as a colleague pointed out: It looks very nice, but what is the work of art? What are you trying to say?
II. SAMPLING AND MAPPING
Where the coding of the video keyboard constitutes the programmatic architecture of the instrument, sampling and mapping represent a compositional process of reviewing the material, outlining sections, cutting, sampling, organizing and cataloguing the material. A process where hours of material is condensed into sets of 88 key "events" that are mapped onto the 88 keys of the digital piano keyboard.
The process involves the following main steps, often jumping back and forth between them along with making continuous adjustments in the program coding:
1. Reviewing the documented video material;
2. Choosing between camera angles;
3. Selecting sequences from which to sample;
4. Selecting sample approach (micro sampling, random cut up, single attacks or gestures etc.)
5. Organizing in registers / folders (located around specific techniques, gestures and sounds);
6. Creating keyboard sets by mapping the chosen samples onto the digital piano keyboard (one sample per key, 88 keys in total);
7. Testing in practice and performance.
III. DISSEMINATION AND PERFORMANCE
Over time, the video archive grew to become an extensive reservoir of material that covered the various playing approaches, at different locations, in different light settings, on different pianos (old and new), at multiple angles, each revealing a different version of the recorded action. A reservoir of audio-visual percepts yet to be experienced anew in the performative and presentational context of the video keyboard.
Every performance on the video keyboard seemed to suggest new approaches: playing blindfolded in front of an audience, using it in ensemble improvisations solely as an audio instrument, performing at a documentary film festival (CPH:DOX), etc.
Towards the end of the project, the performative use of the video keyboard began to open up ideas about a multimodal form of dissemination. Any kind of visual and auditive content could be programmed and mapped to the keyboard: texts, diary notes, quotes, graphics, pictures, along with other excerpts from the research archive. This pointed towards a new research project entitled Interferences, that investigates the potentials in-between dissemination and performance, diving deeper into the intermediary space which the video keyboard had opened up.
As the relations between player and instrument changes in the setting of the video keyboard, it also brings the question of memory in improvisation to the forefront, as one is playing (and being played by) earlier versions of one self.
The relational thus becomes core the work. In a letter to the project supervisor, Bjørn Kruse, I touch upon this, where a set of twelve cardinal points all played an important part in developing the video keyboard: both in the coding stages and in the performative practice.
What seems to be one of the most important discoveries in the project so far, arises from an attention to dynamic relations, where the relational itself is seen as a dynamism, an on-going oscillation. Up until this point, the following set of relations consisting of twelve cardinal points, have constituted in the practice. These are relations between [14]:
repetition – difference
motion – rest
speed – slowness
material – force
affect(ing) – affect(ed)
virtual – actual
These connect among and across themselves on various levels, conceptually, methodologically, practically, performatively, in the process of my research practice. And this attention to the relational, from the single strike of a note to the overall movements of the project has gathered my activities and given them ground. Whenever I feel stuck, in the sense of being ”solo”, alone, I look for relations.
On your right is an extract from a workshop presentation at The Norwegian Academy of Music in Oslo and a performance sound check at RMC in Copenhagen.
Even if I was only about half way through my project, my dear colleague touched on something I had already sensed as a coming challenge, and I began to question the purpose of these documentations, as I could not help noticing their apparent lack of expressivity. Was it the playing that was missing something or was it due to the retrospective nature of the documentation?
In a diary note of an earlier date, before the multi-camera session, I had to already grasped this problem in questioning the intricate balances between the private, the personal and the performative:
It seems to me that the material is more a document of a private session, an inquiry or a practice, than it has the performative qualities of a work of art. The question is of course where I find myself in that balance? Between the private, personal and performative? The material does not seem to communicate outside of the four walls of this studio. As a listener, I need to bend into what's going on, listen into that room, because the room and what is going on in it, is not really speaking out. It is almost as if it doesn’t even try to.
So even if it has been a relevant step to sit "privately" behind safe walls, and document my activities, I am wondering about how to make those sessions of documentation more relevant in how they relate to a surrounding context, an audience, as in a performative setting. This is not to say that I’m not experiencing any relevance in the process, not at all. It is about working with other orientations in the documentation process: to also play out, beyond this situation I am sitting in.
These reflections grew to become key questions in relation to how (and if) the research practice would condense in performances and released works. Was there a way to change the retrospective values of the recordings? To flip the perspective from observation/documentation to expression/performance, by letting the recordings become expressive means themselves? Again, the question was, if there was a way to re-ignite the documentations in a more playful and performative domain? A way of mediating a live experience that would converge the documented past(s) in a playing presence?
This also pointed to a lack on intention, of reaching out to the viewer and listener. As a viewer I was being told or shown what to hear, creating a gap between viewer and the watched. I was simply lacking a dimension of imagination, a complexity of the imagery and soundscape which would activate, or even inspire, my own interpretation and (inner) experience.
When bringing these questions up midway into the research project during a talk at the Rhythmic Music Conservatory in Copenhagen, something unexpected happened. The musician, composer and programmer, Gianluca Elia, whom at the time was studying at RMC on the master's programme, responded immediately by offering his competencies as a programmer. Suddenly, in a leap from question to response, this new and unexpected collaboration launched the development of the video keyboard.
One approach was to think in terms of a timeline, simply by mapping samples conceived for the beginning of a performance to the lower keys and going all the way to the top for the ending. This was also a way to map and score a narrative moving from one section of samples to another. Nonetheless this mapping was still possible to approach in other ways than linearly, for example starting from the middle of keyboard and going in contrary motions or in other ways to create playful intersections.
The examples below illustrate the transition from documentations made prior to the Video Keyboard to versions performed on the new instrument. Moving from left to right, the sample is first shown in its original form, then cut up into fragments (micro samples) that are distributed across the digital keyboard, followed by a video where both the original sample and the micro samples are played simultaneously, superimposed on each other, unfolding a quite different interpretation of the initial motion.
From motion photography to micro sampling
The action photography Étienne-Jules Maray and Edward Muybridge also offers a contextual linkage as to how the micro sampling is approached visually. Rather than searching for sonic bits within a sequence, it was the visual fragments of a gesture that caught my imagintation. To cut out and play around with the frozen images as a way of remodelling and stretching the continuity of an otherwise simple gesture.
There is obviously an enormous technological leap from Maray and Muybridge to present day audio-visual programming possibilities, but the idea to (re)capture and study bodily movement, whether for scientific reasons or as aesthetical expressions (Duschamps' descending nude not forgotten) is a vital thread in both my own practice and the references here mentioned.
The following example illustrates micro samples of a longer duration. Now not as stills but frozen video sequences of microscopic frequences of (repeating) motion.
Cataloguing indexes and their subsequent sample sections are based on a somewhat loose terminology according various to practice-oriented aspects: camera angles, playing approaches (i.e. stickworks, footworks, fieldworks and more specific gestural actions such as arpeggio, glissando, single attaks/hits etc); "non-playing" actions (intermissionary, preparatory and transitive actions such as entering and leaving the frame, sitting quietly, walking around the piano etc.); bodily gestures, sample techniques (i.e. micro sampling, random cut-ups etc.); sample durations etc. subgrouped into registers/sections that are used in the sample sets.
Even if every section within a sample set is based on a particular perspective, the content of these indexes and subsequent sections have overlapping characteristics.
Steps of coding, testing, trying
The first step was to agree on the key concepts of multilayeredness (both sonic and visual) and basic instrumental parameters, before then translating it into coding language (C++ and SuperCollider). In a text about the programming work, Gianluca reflects:
"... the idea is to turn documents into material for performance, by changing the action of displaying a reproduction into that of producing and projecting a sound-image, putting back into play that real-time sensitivity that generated them. It is a subversion of the idea itself of documentation, a shift of point of view that requires a different relationship and responsibility to the means involved.
The first point of contact between Søren and the video keyboard is the keyboard itself, which is a digital keyboard, or simply, a MIDI controller.
While the research involves, for a great part, the physical relationship between the player, the instrument and the space around them, this instrument brings him into the digital field, where the medium doesn’t transfer mechanical energy anymore, but converts it to information, signals that trigger the display of a video.
A lot of the atomic sensitivity and connection that SK experiences as a piano player was initially lost, and in part was completely impossible to re-establish, because of the way the instrument was designed.
Quickly confronting the two different medias of expression, the piano and the video keyboard, we find that they share the keyboard structure and interface, but they differ greatly on the effect of pressing a key. They are both means, instruments, and they transfer the physical energy of the player to the activity of another object (key-hammer-string or key-signal-video)."
Testing the first beta version of the video keyboard sometime in mid July 2017, I experienced entirely different relations of play (player-instrument, player-played) than what I did at the piano. Even if the touch on the digital keybaord was much less intimate (as if playing a digital keyboard or synthesizer), the resulting sound-image affected my response in unexpected ways, let alone the fact, that I was both watching, hearinga dn playing with footage of myself. The acoustics of the piano were amplified on a speaker systemt and the video image further supported that amplification. The shift from passive retrospection to playful interaction suddenly began to fire up the documented. It was an important teaching as to how a performance practice, almost of any kind, needs an audience for it to really gain insight on the forces one is working with.
By cutting, sampling and remapping audio-video files to the 88 keys of the digital keyboard, bits and scraps of documentation showed different events, situated at different points in time. These could be played and layered in multiple ways creating not only images, but also sounds that were humanly impossible from a (solo)pianistic point of view. One key could represent an entire pianistic action. When layered with other keys it expanded the concept of multilayeredness into new sonic territory.
The example above illustrates a different sample approach (and camera angle). Here, a recorded improvisation (GoPro camera set up in my working studio) is cut up randomly by using a program specifically coded for the project. The subsequent video keyboard version is an example of how I (re)play and improvise anew on the cut up material from the original improvisation. A layering of play, which adds further nuance to the multilayeredness involved in solo playing and opens (at least) a double fold, a duologue: there's not one but two players: the virtual player (re)presented in the samples and the physically situated player who unfolds these in a live performance. Again, it is important to stress that the video keyboard track above is a 1:1 representation of the improvisation in the unedited form. Any manipulation of the samples, such as looping, stuttering, speeding up and slowing down, is done during the improvisation.
Knowing that the programming would open up for infinite ways in which to manipulate the material, we made a clear set of dogmas for the digital processing according to different parameters.
Gianluca explains here two of the main features:
1. Dynamic affection of speed on multiple layers;
Mapping the pressure a player puts on each key to the playback speed of the triggered video. This was needed to provide a way to dynamically set different speeds for different videos, to construct assemblages of different speeds at the same time, and opened up for a whole chapter of manipulation of visuals and sounds that brought us even further away from the pitch field of the well-tempered piano.
2. Pixel addition and multiplication;
When playing more than one video at the same time, my first choice was to use transparency. However, we found other ways to blend pictures together, by adding or multiplying the overlaying pixels.
Through the use of this technique, it was even more evident that the multiplicity of videos was leaving traces. It was not just a matter of juxtapositions, but more and more a multiplicity in the act of becoming one; a blending, where each part influences each other and the whole, as opposed to a stack of independent layers.
The examples below show the shift from an original footage of a glissando to a video keyboard version in which I play around with the speed of the glissando along with overlayering samples in different velocites.
The video keyboard could be said to open a third acoustic region that conjugates the well-tempered and un-pitched sounds of the acoustic piano into a flexible pitch field.
The function of the acoustic piano pedals (una corda and sustain) were also translated to the video keyboard pedals in order to create sustain and loops of varying duration (from looping a full sequence to the stutter of an instant).
As a basic limitation in the programming work, we decided that the instrument could only manipulate the concrete, raw information granted in the audio and video files. No added effects, such as reverb, delay, distortion, compression etc.
The video keyboard features were not planned from the beginning. They came out of a continuoust trial process across various stages. One of these stages, both processual and performative, occurred during an artist’s residency at Mandagsklubben, held at the experimental music venue, 5E in Copenhagen. On four consecutive Mondays, we beta tested the video keyboard during live performances. To perform live was a radical choice, since the instrument was still very much in the making. However, I thought the process needed to be radicalized in more than one sense: Firstly, to speed up the coding and beta testing, which also sped up the sampling process; secondly, to test the instrument live in a room with an audience, to get more realistic input on how it communicated; thirdly, as the following chapter explains, to explore different sampling and mapping approaches, by creating new sample sets without repeating any of the material from a previous concert. This required continuous revisits to the growing archive of video documentations.
Right pedal:sustain (loops any given sample in full length);
Left pedal: loops any given sample to an adjustable duration down to a single frame, duration 1/25 of a second. (all footage is recorded in 25 frames per second)
USB controller designated to control the program (shifting between settings, frames, pedal settings, activating/deactivating various parameters, turning/scaling up and down volume, light, speed, color etc.)
Keyboard mapping of the audio-visual content. Each strip of tape represents a set of 88 samples giving an overview of the digital content of the 88 keys.
Noget om
- SAMPLING:
single actions: movement/phrase/gesture
single hits: tonal, chordal, non-tempered, percussive
intermissionaries/transients
samples cut out from a continuity (say a 5 minute improv cut into 20 samples)
micro samples (cut out from a single gesture/movement)
- MAPPING (keyboard-mapping, composition, "fieldworks", mappe og spille lineært, kromatisk eller kontrapunktisk/modsatrettet eller frit mellem GROUPINGS)
- Multilayeredness of:
sound+image;
sample/representation+live presence;
sonically->temperaments/tuning (flexi-tempered, well-
tempered, non-tempered),
visuallly: gestures, body movements
rhythms (both sonically and visually)
replication/repetition (same action repeated, frozen, added and same person/body layered)
layering of diff chronologies, situations, pianos
Context: Simon Steen-Andersen, Rendered, Run Time Error, Piano Concerto, Cello study (image projection)
+ Conlon Nancarrow (machinic activation of the piano mechanics -> speeds, textures, rhythms, co-ordinations impossible for a (solo)pianist).
Noget om the presentational hvor jeg live vælger ca. samme opsætning som Rendered, altså med ryggen til - dog mere i direkte kontakt/samspil med det jeg ser. Rendered er en live projektion af det der foregår, live in action, hvor mit er en interaktion mellem a player both present and represented in various forms, perspectives, speeds, where different temporalities are conjugated into a more ambigious, polychronic experience.