Audiovisual

To be able to create an audiovisual performance, it is of great value to have a good and thorough understanding of what audiovisual performances entail especially in the context of classical music. This consists of looking into conducted research regarding this topic, including the reason for the increasing number of audiovisual performances in the last decennia, and by looking into scientific research on the visualization of music. This will be done by sketching a historical context, and by looking into the elements of color and synchronization. I would like to emphasize that this chapter only touches the surface of research, as it aims to inspire, rather than to set up an extensive scientific foundation. 

Audience decline for classical music concerts [11]

The increasing number of audiovisual performances in the classical music scene goes hand in hand with the decline of audiences for concerts where classical music is being played. Research states multiple reasons for this decline in audience. Firstly, classical music concerts have become a “predictable experience because of formulaic programming strategies” (Idema, 2012 as cited by Sun, 2017), and young adults miss engagement due to the classical concert rituals (Richards, Marques, & Mein, 2015). “The current system provides passive experiences, while millennials prefer to actively participate in concerts” (Sun, 2017). But others mention it is “not only the cultural status and significance of classical music but, more radically, the very idea of what classical music is” (Pontra, 2022). They call for research to understand how people think about classical music and how those ideas are formed. Adding the following quote to this statement: “The low attendance within Classical music auditoriums proves that the current ways of dissemination and communication of Classical music events are inadequate to cope with the trends of the modern era and thus, the majority of young people do not participate in them” (Hambersin, 2017). Therefore, it is also a matter of having the right marketing and PR. Other research mentions that “going to classical music concerts relies on the satisfaction of social needs, and not having somebody to go along presents a relevant barrier for attendance [of younger people aged 18-34]”, (Baker, 2000, Roose, 2008 & Michael, 2017 as cited in Merino Ruiz, 2020). Another reason in terms of social expectations is that: “[There are] people who do not attend classical music concerts at theatres or halls because they feel like they don’t fit in with the expected code and related class level” (Bendana Rivas, 2019). Lastly, “Surveys show audiences thirsty for cultural experiences but for whom the raw exposure to musical works is either too unfamiliar or not stimulating enough” (Amaral, 2021). Concluding this paragraph, the audience decline relies on multiple reasons going from the classical concert ritual, and the social experience to marketing, and the expectations of an audience.

So one of the solutions and attempts at attracting (new) audiences, is to make classical music concerts visually attractive and to “create a multisensory experience to replace the traditional way of passively ‘listening to’ a classical concert, [as young adults] gain satisfaction from a concert’s visual, auditory and psychological aspects” (Mielonen, 2003). As the “ocular sense was strengthened significantly, after the growth of digitization” (Forde Thompson, Graham, & Russo, 2005), “people in general, and especially the younger generation, are habituated to the visual experience” (Greckel, 2021). Not only do “audience[s] with no music education rely more on ocular signals rather than aural ones, also visuals within music experiences are valuable and helpful to professional musicians” (Chan, 2010). Professional musicians and ensembles search for ways to think beyond interpretation and the musical work, as they are “no longer satisfied with merely executing scores” (Amaral, 2021). So in conclusion, creating a multisensory experience through for example audiovisual performances can help attract (new) audiences to classical music concerts. As the audience is declining, it is a natural conclusion to therefore look for other possibilities for portraying classical music. It answers why there are (or should be) so many audiovisual performances rising in the last decennia, as well as possibilities for funding [12] these types of projects[13].

Dr. Wil Greckel accurately describes the situation we are currently facing: "Trained musicians are capable of enjoying the most complex music in an unadorned, abstract form, and they most often prefer this type of performance. However, if the decline of audiences in our concert halls is to be taken seriously, we cannot afford to go on assuming that the general public, and especially the youth of today, is always able and ready to enjoy classical music in such "pure" and abstract terms, and in an atmosphere of nineteenth-century formality. If the addition of visual elements to our performance communicates the spirit of the music more vividly and brings audiences back to the concert halls, then the increased use of visualization must be given important consideration." (Greckel, 2021)

Visualization of Classical Music

The previous paragraph concluded that the visualization of music can add satisfaction to the experience of (classical music) concerts. This paragraph goes into the more practical side of this topic. First by providing a historical context, then by looking into conducted research regarding music visualization, including different ways of using musical narrative for visualization.  

Historical context

Visualization of classical music, or the connections between art and classical music, has been around for a long time in different forms. As Professor of Music Education and Music Psychology Friedrich Platz and Doctor of Philosophy and Professor of Music Psychology Reinhard Kopiez mention in their research on audiovisual presentation: “The visual component is not a marginal phenomenon in music perception, but an important factor in the communication of meaning” (Platz, & Kopiez, 2012). In historical context: “Visualization was a matter of connecting auditory and visual perceptions, with inspiration drawn from fine art and music. It is documented that artists in every historical period tried to combine music and visual elements into one meaningful ensemble” (Novotna & Ašenbrenerová, 2021). A good example of how music and art are closely connected is composer Modest Mussorgsky’s work Pictures at an Exhibition, which he wrote based on the pictures by Viktor Hartmann. Or the other way around: painter Wassily Kandinsky was inspired by music for his paintings and believed that “painting might aspire to the abstract condition of music”. When describing his experience going to Richard Wagner’s opera Lohengrin: “I saw all my colours in spirit, before my eyes. Wild almost crazy lines were sketched in front of me.” (Kemp & Blakemore, 2006). Next to Kandinsky, painters like Piet Mondriaan, Jackson Pollock, and Mark Rothko all “sought to transfer musical parameters, rhythm, form, and texture into their visual language” (Basica, 2021). Artists who have also explored the way of “transposing musical structures and behaviors into moving image” (Basica, 2021), are abstract filmmakers such as Walther Ruttmann, Hans Richter, or Oskar Fischinger. They have created and experimented with visual music: “Time-based visual imagery that […] can be accompanied by sound but can also be silent” (Evans, 2005). They are united by the idea that “visual art can aspire to the dynamic and nonobjective qualities of music”[14] (Mattis, 2005), and they sought a visual equivalent to music which could include “programmatic randomness, repeated layering, poetic structure (a visual limerick), repetition and evolution” (Taberham, 2018). These abstract films combined music and visuals seamlessly. At first, the works consisted of shapes and patterns moving in time with music, later the abstract shapes “started being choreographed to classical music and other types of music” (Basica, 2021). Taking Oskar Fischinger as an example, who had put an excerpt of J.S. Bach’s Brandenburg Concerto no. 3 to an experimental film of abstract paints, called Motion Paintings. His work “visually reflect[s] the performance qualities of music through form, dynamics and pattern” (Sun, 2017). Romanian composer Constantin Basica mentions in his lecture on audiovisual performances that “visual music is perhaps the ancestor of all audiovisual practices today (…) [as it] existed through the intersections between music and the visual arts”. Another example of visualization inspired by classical music, but a more popular example, is Walt Disney’s Fantasia from 1940 (Rudenko, McDonnell, Layden, & Haahr, 2022).

Next to this history, “there was also the parallel development of music as purely an aural art-music as an art for the ears only, not for the eyes” (Greckel, 2021). As Dr. Wil Greckel mentions in his article: “For it is during this period of musical history that one sees the ultimate visualization of music in the emergence of opera and ballet as major art forms, and, at the opposite end, the full development of absolute, abstract, or "pure" instrumental music.” (Greckel, 2021). This intellectualization of music reached its peak in the classical tradition of Haydn, Mozart, and Beethoven, where music was only about the music itself. In the 19^th century “program music” gained popularity, which suggests visual images through descriptive titles, and the listeners imagined the visuals instead seeing them. Well-known examples are Sergej Prokofiev’s Peter and the Wolf and Camille Saint-Saëns’ Le Carnaval des Animaux, were both composers who sought to combine music and (imaginative) image (Novotna & Ašenbrenerová, 2021).

Examples of Music Visualization

Researcher Wing-Yi Chang (The Hong Kong University of Science and Technology) mentions that to elevate an audience’s concert experience, visualization systems can be used to show or highlight artistic elements of classical music. This results in an increase in the audience’s enjoyment of the music (Chang, 2010). In Visualization of Music and its Application in the Process of Education by Doctor in Biochemistry Eva Novotna and Doctor in Pedagogy, Ivana Ašenbrenerová talks about using visualization of music in education. The goal is therefore different than the previous example, but they do mention three principles required for the visualization to fulfill its purpose: “1. Visualization must not distract (…) from the very essence of music, 2. Visualization must result in a semantic and structural analysis of the musical work, and 3. Visualization is the initial experience triggering the child’s creative activity” (Novotna, & Ašenbrenerová, 2021). As Dr. Wil Greckel also mentions in his article, when looking for a classical work to visualize, it is probably easiest to look into works considered program music, than for example the music in the classical tradition of Haydn or Mozart. “Yet, ballet performances in the past 50 years or so demonstrate that the visualization of abstract or absolute music is as successful and prevalent as with program music. (…) [So] there is really no type of musical work which should be arbitrarily excluded when one is considering the use of visual elements.” (Greckel, 2021). This brings up the question of how to accomplish this, keeping in mind that “the complexity of classical music and sensitive content of intellectual emotion[15] makes visualization approaches intriguing but raises the question: What approaches to music visualization are suitable for music narratives, i.e., which approaches engage in a suitable fashion with music plot and dramaturgy and still remain true to musicology?” (Rudenko, McDonnell, Layden, & Haahr, 2022).

In Exploring Classical Music Narrative through Multimodality in AR/VR Experience, research by pianist and Doctor of Arts Svetlana Rudenko, she mentions that classical music narratives can be analyzed in different ways: by focusing on the musical structure (like a sonata form with contrasting themes), emotional narrative (like a Prelude, capturing a specific mood), or a program narrative where the storyline is set by the composer, such as Berlioz’s Symphonie Fantastique. (Rudenko, et al., 2022) By mentioning one example and two experiments from this conducted research, a few different approaches to looking at musical narrative as a starting point for visualization will be shown:

Some examples of audiovisual performances using lights, that have all in some way used different kinds of narrative inspiration for the visuals:

• Musical structure/interactivity: Harpist Doriene Marselje created In a Landscape (Marselje, 2021), an audiovisual experience with a modular light installation reacting to the harp, where lights surrounding the harp would light up whenever she plays a certain frequency. So, everything that is played can be seen directly.
• Emotional narrative/mood evocation: Blauwe Uur who designs light projections filling an entire space, in collaboration with harpist Remy van Kesteren (Kesteren, 2018). This results in an audience being fully immersed into Remy’s world, but still leaving room for wandering the mind and imagination.
• Program narrative: Performance of Julius Eastman’s Gay Guerrilla during the Re:ply to All Festival, where the piece was performed, together with a light show designed by Boris Acket. Here the scenography influenced the experience of the work, and the lights told a visual story (Basilova, & Cheng, 2023).

It is important to take into consideration that the use of music as a narrative medium is quite complex, more “than it may appear at first glance. (…) In classical music compositions, music narrative tends to be complex and symbolical and can be challenging unless the composer supplies a written program” (Rudenko, McDonnell, Layden, & Haahr, 2022). When starting the process of visualizing a piece of music, it is important to keep in mind what kind of narrative the visualization is based on. Three possible options can be an existing narrative, a certain mood, or the musical structure. In the case of program music, there is already inspiration for visualization in the program notes or title, in more absolute music, the visualization can be looked for in the music. At least when the aim is to let the music be enhanced by the visuals.

Crossmodal correspondences

It is meaningful to know about the correlation between seeing and hearing, or what is known as crossmodal correspondences [16] (without going too deeply into the biological side of this matter). From here on a better understanding of audiovisual performances and a theoretical framework are created, to inspire the audiovisual experimentations done in this research. This will be done by looking into the topics of synchronization and color; two big topics in audiovisual perception and music visualization.

Synchronization

Psychologically, hearing and seeing work differently in our brains: they are two distinct pathways for understanding our environment[17] (Schacher, 2008). People who see and hear usually have a “multimodal perception of ‘audiovisual phenomena’ in the world”. This is because we perceive events using a mixture of senses (Basica, 2021). Whenever we perceive this information through different modalities, such as in this case, the hearing and the seeing, we do look for a coloration. This is especially the case when images and sounds are synchronized. As Dr. Paul Taberham (PhD in Film Studies) mentions in his book Lessons in Perception, The Avant-Garde Filmmaker as Practical Psychologist: “Our sensory systems are compelled to perceive synchronized images and sounds as part of the same event – psychological evolution instructs us to cross-check modalities to confirm the veridicality of our perceptions." (Taberham, 2018). Translating this to music: “If the patterns and rhythms are confirmed across modalities, the information carried by the sound and image is perceived as a single event” (Gibson, 1969), or in other words: if what we hear, synchronizes with what we see, we naturally combine the two in our system. When this is exaggerated, as sometimes used in film, there’s a term for this kind of synchronization, namely Mickey Mousing: where “the film score illustrates and accentuates and in its expression is closely related to the images and the plot” (Beller, 2011). Or as Romanian composer Constantin Basica calls it “music that attempts to strictly imitate the visual action” (Basica, 2021). In the same lecture, Basica references to Statement on Sound (Eisenstein, Podovkin, & Alexandrov, 2014), which states that “sound works either in parallel with or in counterpoint to the visuals.” Where parallel means the sound voicing the image, and counterpoint means that there is a disconnection between the seeing and the hearing, or as experimental composer and theorist Michel Chion calls it “audiovisual dissonance” (Chion, 1994 as cited in Basica, 2021). These alternative sound-image relationships have been among others explored by visual music makers. They concluded that synchronization helped to broaden the appeal[18] of their films (Taberham, 2018). Next to that, symmetry also made their films more appealing to general audiences[19]. When looking into synchronization between the aural and the visual, an easy musical parameter to think of could be rhythm: where patterns in sound will align with patterns in light design, visual art, or film loops. Other parameters could be the combination of the visual dimensions of 2D and 3D with stereo or mono sound (DeWitt, 1987), or, as mentioned by computer animation pioneer John Whitney, the combination of consonant and dissonant harmonies, and visual order and disorder (Whitney, 1987).

Color

Dr. Wil Greckel mentions that “Color association with pitch is apparently quite common, but most often not associations with specific pitches, but rather with a range of pitches, as in dark colors with low pitches and bright colors with high pitches. Another general category of ‘color hearing’ involves the association of colors with tonalities”[22] (Greckel, 2021). Composers like Scriabin, Schoenberg, Cage, Penderecki, Ligeti, and Messiaen used a color/sound relationship as a tool of expression of musical notation (Poast, 2000). This can result in color music: music “intended for instrumental practice in conjunction with [a] simultaneous projection of changing colors onto a screen”, as mentioned in Encyclopedia Britannica[23]. One (famous) example is the work Prometheus by composer Aleksandr Scriabin, which included a Clavier à lumières[24], an instrument that did not exist at the time, but would have provided the music with colored light to make the “harmony moves more evident” (Rudenko, McDonnell, Layden, & Haahr, 2022). This work comes from his perception of hearing/seeing sound as literal color, a perceptual phenomenon which is called “Synesthesia”: “A condition in which someone experiences things through their senses in an unusual way, for example by experiencing a colour as a sound, or a number as a position in space” according to the Cambridge Advanced Learner’s Dictionary & Thesaurus. Some call this form of synesthesia “colored hearing or chromesthesia”’ (Ward, Huckstep, & Tsakanikos, 2006). Scriabin believed that “the integration of colored light within a symphonic work would act as a ‘powerful psychological resonator for the listener”(Peacock, 1985)[25]. Visual music (as mentioned in the previous paragraph Visualization of (classical) music) is according to Constantin Basica linked to synesthesia as it attempts to “recreate the types of crossmodal connections between sight and hearing experienced by some synesthetes artists” (Basica, 2021).

As color/sound associations occur and stay stable within individuals with synesthesia (Baron-Cohen, Burt, Smith-Laittan, Harrison, & Bolton, 1996), the same tones or grapheme do not “necessarily evoke the same color in different people” (Brang & Ramachandran, 2011). Next to that, for the majority of people without synesthesia, the emotional mediation explanation is most likely the reason for the connection they feel between musical sounds and color (Spence & Di Stefano, 2022). Non-synesthetic people do experience crossmodal correspondences, but these are seemingly random, as said by psychologist Professor Charles Spence (Spence, 2022 as cited in Rudenko, 2022). According to psychologist and professor Bernard I. Levy, ‘cool’ colors are soothing, and ‘warm’ colors stimulate (Levy, 1984). Research by Dr. Thomas Clarke and Professor of Psychology Alan Costall agrees, but adds that their study “raised new issues concerning color as emotionally multi-faceted, individual differences and experience, and the specific influences of culture upon emotional reactions” (Clarke & Costall, 2008). Researcher in psychology William Griscom then concludes that people (with and without synesthesia) tend to pick similar colors for the emotions associated with different sounds, probably relating to deeper connections in the brain related to feeling and understanding (Griscom, 2014). Katleen Karlsen (Master of Arts in Humanities) mentions in her research The color-music connection: philosophical, aesthetic and scientific perspectives, that a direct connection between color and sound exists on an intuitive and aesthetic level supported by philosophical and historical evidence, “although scientific proof of a direct connection between color and sound remains subjective and elusive.” (Karlsen, 2001)

Synchronization of the visual and the aural is a powerful way to appeal to the audiovisual work to the audience. This can be done in different ways by looking at correlations between sound and image, taking into consideration that visual information also has musical qualities. This can suggest possibilities for audiovisual performance in what kind of visual material can be used in combination with certain musical parameters, for example by combining dynamics with light intensity or letting the lights move with the flow of the music. In conclusion, combining sound with color has inspired many researchers, composers, and scientists. They agreed that there is a relationship between color and sound/music, although it is individualistic through, for example, synesthesia or an emotional correlation. It can be taken into consideration that the majority of people can have the same associations.