In earlier chapters, this research has created a theoretical framework for interdisciplinary collaboration, for music visualization and on Paul Hindemith’s harp sonata. It has provided narrative information to inspire the visualization of the sonata, which will be done in collaboration with creative art director Sebastiaan Smink, as mentioned in the introduction. This chapter describes the collaborative preparation of making an audiovisual performance and shows the conducted thought experiments.
A few things considered: as mentioned earlier, Hindemith’s harp sonata is not a form of program music, so the narrative had to be looked at differently. Next to that, our goal is to create an enhancement of the music, as has been mentioned in the visualization paragraph, and not to create a counterpoint, or “audiovisual dissonance” as mentioned in the crossmodal correspondences paragraph. The last thing mentioned is that Sebastiaan and I are both not synesthetes, so sound/color correlations will not be based on synesthesia, but more on emotional correlations.
The visual development phase refers to the process of developing visual concepts and designs. This includes cooperatively discussing ideas, using imagination, making mood boards, and checking the practicality of ideas (including financial practicality). As mentioned in the introduction, the goal is to make an audiovisual performance with the use of lights. From the (narrative) analysis of the sonata and the music visualization chapter, a few different (visual) ideas came to mind, which have been used as a starting point for the thought experiments.
The first idea originated from the works of “visual music” makers such as Viktor Eggeling[30], Walther Ruttmann,[31] and Oskar Fischinger[32], who visualized the music through shapes and patterns that “reflect the performance qualities of music through form, dynamics and pattern” (Sun, 2017), as mentioned in the audiovisual chapter. As these ideas are about shapes, I thought about the idea of using the ‘Urmotiv’ as a shape, form, or unit, which would stand for this motive. As mentioned in the chapter on the harp sonata, the ‘Urmotiv’ appears throughout the sonata in diverse ways (rotated, inverted, retrograde), so the visual unity could do this as well, as the shapes did in the work of Viktor Eggeling. The visual result would be a synchronized visual representation of the microstructure of the piece, which could have been projected on the harp, in the room, or on a screen.
This way of visualization raised a question of practicality in a few diverse ways. If the performance is a live performance, there would be three options for synchronizing the music with the visuals. 1. Performing with a click track, 2. Performing with an annotated score with a timer (as was done in the performance of Julius Eastmans’ Gay Guerilla), or 3. Someone reading the score while giving cues to change projections. A click track would have given little space for playing with musical decisions and phrasing during the performance, and an annotated score with a timer would not have worked as there is too much musical information (contrary to the minimal music of Eastman). Otherwise, we needed to have a third person involved as Sebastiaan could not read the score that well. Another question was raised with this idea based on the motive. Keeping in mind the key qualities of interdisciplinary collaboration as mentioned in the introduction, this idea did not provide “Common Ground”[33], as it is based on the analysis of the score, which would have put too much emphasis on the aural discipline instead of the visual discipline. Also, it did not provoke a “Shared Creative Vision”[34] as Sebastiaan’s interest lay in scenography, and not in animation design.
The second idea is inspired by the visualization of Alexander Scriabin’s Piano Sonata (as mentioned in the visualization chapter). The idea is based on the musical structure and the archetypes of musical texture, combined with the narrative analysis by Jackson Sliva as mentioned in the Hindemith chapter. Instead of giving shape to the “Urmotiv” as mentioned in the previously mentioned idea, it would name the different themes in the music, moving along and changing over time. Visually I got inspired by Thomas Wilfred’s Lumia Suite, op. 158, a work with the lights gradually changing over time. The musical themes would be defined by different colors of light, changing over time, overlapping each other, and coming back in varied ways. It would make the musical structure visually clear and will use the lights as a narrative tool, instead of just being an accompaniment.
This second idea (in combination with what is laid out in the next sub-paragraph) turned out to be the basis of the final performance, as will be mentioned in the following chapter Live Experimentation. Different sections of music would be defined by different colors of light, with the music going hand-in-hand with the visuals through interactivity, and by playing with light intensity and speed. In terms of interdisciplinary collaboration, Sebastiaan and I both shared this creative vision, and would both learn about this process, inspire each other, and would be able to brainstorm about this idea[35].
The final idea is not based on the structural analysis of the piece, but on the atmosphere/mood of the work, as has been done in Johannes Brahms’ Intermezzo, as mentioned in the visualization chapter. For the harp sonata, this idea is about the narrative ideas on the sonata as mentioned by among others harpists Marcel Grandjany and Edward Witsenburg, with the first movement being based on the image of “a cathedral, majestic and beautiful. In the end, the shadows grow longer and longer, the sun descending at the end of an afternoon”[36]. Visually this idea could turn into a light installation. It would be based on the cathedral windows or the shape of a cathedral, the graveyard, and the moving light, portraying the atmosphere of the different movements of the sonata by using different colors of light. In combination with the second idea, this third idea also turned out to be a basis for the final performance, as will be explained further in the following chapter Live experimentation. This idea was also the best one in terms of interdisciplinary collaboration as it would be beneficial for the mutual transformation, it entailed a shared creative vision (as Sebastiaan’s interest lies in scenography) and it would touch common ground, as we mutually understand and share the values created by this idea.
From the visual development phase, we started our thought experiments, mostly inspired by the second and third ideas: a visualization based on the macrostructure and the narrative context of Hindemith’s Sonate für Harfe. Sebastiaan made two mood boards: one inspired by a ray of light, the other inspired by a stained-glass window. From here the process of “Design Thinking” started, which in practice results in evaluating every step, going back to the first idea, checking if the design is still on the right track, and reflecting on every choice made throughout the process (Dam, & Teo, 2022)[37]. This process resulted in a few ideas.
Sculpture inspired by cathedral windows: We came up with the idea of a window-like sculpture, inspired by the idea of a cathedral window. Light would come through this window, creating different atmospheres and shadows in the performance space. Sebastiaan created AI-generated images[38] through Blender, to help us shape our ideas visually.
A ray of light: A ray of light, like a sundial or a door that slowly opens, moves around the room. An idea coming from the historical context and the turbulent time in which Hindemith wrote his harp sonata, and from the theme of death coming from the third movement. The theme would be the passing of time and eventually death.
Sculpture with milk glass and LEDs: This idea merged from ‘Sculpture inspired by cathedral windows’ but would be made from milk glass and LEDs mounted on the inside of the window-like shapes, which would create the effect of stained-glass windows. The various parts of the window-like structure would slowly light up and would in the end turn out to be a visual story. This idea was not used, as the properties of the milk glass would not lend themselves well to transferring the light, as concluded after material experiments from Sebastiaan.
Cubes: This idea appeared from the failed material experiments of the earlier idea, which led to thoughts about practicality. Sebastiaan mentioned the new idea of making a scalable installation in the form of cubes, as the budget was not yet clear, and there was little time left to rethink the earlier idea. These cubes, made of LEDs, could move around and could be made from 4 to even 20 cubes, depending on the budget. The cubes could be used in different settings or other projects. Within this performance, they would be used to create different atmospheres in the space, by using distinct colors and by lighting up different sides of the objects. This idea was not used as it was too difficult to finish in the given timeframe, plus we did not have the right expertise on our team to make this work. Looking back on it, the idea of cubes had no clear connection with the harp sonata.
New composition: At one point we concluded that the harp sonata wasn’t necessarily working with the idea of an audiovisual work (as will be mentioned in the next chapter). We came up with the idea of making a reconstructed version, using elements of the different movements, and creating a new composition with the use of pedals and presets in Ableton. Hereunder are three of these experiments:
- Hindemith_12.06.23_Test_2.wav: We experimented with different effects through Ableton, while I would rub my finger on the soundboard and glide over the metal bass strings. It starts with the beginning of the first theme of the first movement in a higher register, together with all the rubbing, gliding, and tickling on the soundboard. At 2.10: the violin clef part of the second theme comes in, while around 2.30 the left hand joins. Around 3.10 the descending chords from the end of the second theme of the first movement enter. Throughout the whole experiment, a drone is hearable.
- Hindemith_12.06.23_Test_3.wav: This starts with a free rubato improvisation on the first bars of the first movement with a big delay and a drone lying underneath it. Around 1.20 starts a free interpretation of the development part on the first page of the piece. The harmony then changes at around 2.10 with the drone. At 2.40 a free interpretation of the second theme comes in. After that, random chords, end with a fadeout and a big pedal buzz.
- Hindemith_12.06.23_Test_5.1.wav: This tryout came from experimenting and combining different themes from the second and the first movement. At around 1.00 I started to add notes from the second movement in a different tonality. At around 1.30 I started to add some different themes which clashed. We liked the gamey feeling the beginning of this fragment has, and how, by combining the two themes, a whole new melody was created.
Diptych: From previously mentioned experiments, I concluded that I am by no means a composer, and it was never my intention with this research to compose, or use, and explore harp with electronics. I got in touch with Gijs van der Heijden, who was open to creating a new piece, based on the Hindemith sonata. It would then turn out to become a diptych of the two works, to present and perform at contemporary music festivals. This idea was not used in the end, as there was simply no time for this process[39].
The visual development and thought experiments have resulted in a final idea to try out in the live experimentation phase, as explained in the following chapter: an LED installation, mounted at the beams of the attic space where the performance takes place. Firstly, this idea comes from the thought of using the space itself as an LED installation, instead of putting a stand-alone LED sculptor in the room, as mentioned in the visual thought experiments. Secondly, it comes from the idea of creating the architectural outline of a cathedral with light, inspired by the narrative ideas of the harp sonata. The idea started with having the LEDs go from the floor of the space to the ridge of the attic 6 meters above the floor. This would have created a cathedral-like shape. Due to practical and budget reasons (as we decided to work with pixel-animated LEDs[40], which are more expensive than ‘normal’ LEDs), it has resulted in the shape that can be seen in the video in the next chapter. The LEDs are programmed by Sebastiaan through TouchDesigner and MadMapper and will be controlled by him during the actual performance.