Connecting Embodied Cognition and Sonic Information Design
The modern embodied cognition research program in cognitive science shares some roots with 20th-century European continental philosophy (primarily through Merleau–Ponty). However, it is a distinctive research strand that is more closely identified with cognitive linguistics (Lakoff and Johnson 1980), American pragmatism (Johnson 1987), and more recently with neuroscience (Lakoff 2012). The central thesis of embodied cognition is that the cognitive processes and conceptual systems of the human mind are shaped by the physical and perceptual affordances of the human body (Varela, Thompson and Rosch 1991). When the terms “embodiment” and “embodied cognition” appear in relation to sound, there is sometimes an implicit assumption that for a sound to “be embodied,” it must somehow relate directly to some physical gesture, pattern of movement, or phenomenological observation. Whilst the body is obviously the starting point for embodied cognition research, the final goal is to explain cognition in all of its nuance and variation. Embodied cognition aims to achieve this by rethinking the mind from a bottom–up perspective, departing from bodily experience. In pursuing this aim, embodied cognition, as it has matured as a research theme, has provided a number of theories of how more abstract, higher–level cognition may be conceptualized from a foundational perspective of embodiment. However, it would be a mistake to assume that these theories should refer back to physical experience at every turn or attempt to reduce all aspects of cognition to bodily experience.
From a theoretical perspective, embodied cognition can be seen as arising in response to the identification of a symbol grounding problem (Harnad 1991) within computationalist philosophy, which describes shortcomings in traditional cognitivist treatment of the human mind as a computer in the attempt to adequately describe how mental and perceptual symbols become meaningful (Varela, Thompson and Rosch 1991). The symbol grounding problem points out that the traditional cognitivist/computationalist model of the mind as a computer of mental and perceptual symbols treats mental and perceptual symbols as arbitrary tokens which have no implicit relation to the referents which render them meaningful. We see this replicated in the mapping problem where, as discussed earlier in relation to Grond and Hermann (2012), the sound materials and structures presented in a given sonification may run the risk of being perceived as too arbitrary in their relationship with a given data set, or in which the mapping may even contradict important dynamics within the original data set. The field of embodied cognition in its response to computationalism could be seen as primarily concerned with meaning–making, taking embodied experience to be the site ofenactionfor mental and perceptual symbols. Embodied cognition researchers have theorized a number of novel cognitive faculties that complement the more traditional models of cognition. Developments in embodied cognition have shaped research agendas in a number of important fields related to sonic information design, including computer science and artificial intelligence (Brooks 1991; Dourish 2004), visual perception (Noë 2009), aesthetics (Johnson 2008), music (Godøy 2003, 2006; Zbikowski 2002; Adlington 2003; Brower 2000; Larson 2012; Cox 2001; Leman 2008; Klemmer, Hartmann and Takayama 2006), and HCI (Imaz and Benyon 2007; Hurtienne 2009; Waterworth and Riva 2014; Bødker and Klokmose 2016). Recently, a number of auditory display researchers have begun to apply principles from embodied cognition to auditory display design (Diniz, Deweppe, Demey and Leman 2010, 2012; Dyer, Stapleton and Rodger 2015, 2017; Verona and Peres 2017).
An influential strand within contemporary embodied cognition research is represented in the interrelated theories of image schemata, conceptual metaphors, and conceptual blending, which relate to structural and formal models of how more abstract, logical, or propositional thought may be explained using embodied concepts.
- Image schemata, introduced by Mark Johnson (1987), can be conceptualized as basic building blocks of propositional or relational thought which are derived from physical gestures: commonly shared gestalt patterns of embodied experience derived from sensorimotor experience, providing a commonly shared frame for organizing and enacting meaning and logical consistency to the chaotic patterns of raw perception prior to and independent of conceptualization. As such they provide the basic cognitive building blocks upon which meaning-making, language, reasoning, and imagination are erected.
- Conceptual metaphor was introduced by George Lakoff and Mark Johnson (1980). It is another key embodied cognition theory. Its basic premise is that metaphor is a fundamental cognitive process and that humans make sense of new experiences and concepts using previous experiences and concepts as a frame of reference. A classic example is the “love is a journey” metaphor in which one frames love as a journey, allowing the listener to understand and reason about romantic relationships, drawing upon the concepts and logical structure associated with journeys. As such, a conceptual metaphor is a frame-to-frame mapping “with the roles of the source frame mapping to corresponding roles in the target frame” (Lakoff 2012). The theory postulates that the ultimate ground of all metaphors and frames is found in the image schemata of everyday physical experience.
- Conceptual blending, introduced by Gilles Fauconnier and Mark Turner in 2002, involves “a blending of older concepts to give rise to new emergent properties” (Imaz and Benyon 2007). A conceptual blend involves the integration of two familiar concepts to create a novel concept which contains properties that were not present in either of the two concepts in isolation. The mythical concepts of the Pegasus and the Centaur have been described as blends between the concepts of bird and horse and man and horse, respectively (Martinez et al. 2012). Kendall's (2014) feeling blend, discussed earlier, describes the emergence of novel emotions, meanings, and affective states in electroacoustic music through the blending of familiar everyday sonic materials. One possible fruitful approach to sonic information design, which we wish to highlight here, is defined in Imaz and Benyon's (2007) Designing with Blends. They describe and advocate an approach to HCI design in which systems are analyzed and conceptualized in terms of image schemata, conceptual metaphors, and blends. The design discussed in the following sections is informed by this approach.