Emotional Machines - Composing for Unstable Media.

Thorolf Thuestad

Emotional machines – composing for unstable media. Thorolf Thuestad In partial fulfilment of PhD in artistic research The Grieg Academy, Department of Music, Faculty of Art, Music, and Design, University of Bergen, 2021 The artistic research project Emotional machines - composing for unstable media explores how non-representational kinetic objects may afford affective and emotional responses, in particular experiences of kinship and relation between an audience and animated objects. Further possibilities for artistic expression are opened through composition and dramaturgy, in works where the animated objects perform alongside human performers and other media, such as light and sound. How may such animated non-representational physical objects be developed? How does animating the physical objects change the potential for experience of kinship and relation between audience and object? How does context modulate the potential for experience of kinship and relation between audience and animated physical object? How does the potential experience of kinship with the animated object differ for the creator of the object, and audience? These questions have been investigated through the creation of a set of animated kinetic figures. These figures have performed in a series of artistic presentations such as gallery exhibitions and stage presentations. The artistic research concludes with two complementary public artistic presentations taking place in bergen in June/July 2021; a time-based stage production at Cornerteatret and a gallery exhibition at Lydgalleriet. The presentations are considered connected and have the common title For one – for many – for all. This is an exposition of the research that presents the kinetic figures, the process of developing them, the questions they have contributed to explore, resulting artistic works (with particular emphasis on the final artistic presentations taking place in June/July 2021), the additional artists contributing to these public presentations, and a series of reflections resulting from the research. --------------------------------------------------------------------------- Emotional machines – composing for unstable media Thorolf Thuestad. Denne eksposisjonen presenteres som del av gjennomføring av doktorgrad i kunstnerisk utviklingsarbeid ved Griegakademiet, Institutt for musikk, Fakultet for kunst, musikk og design, Universitetet i Bergen, 2021. Emotional machines - composing for unstable media utforsker hvordan ikke-representative kinetiske objekter kan skape affektive og emosjonelle responser hos mennesker. Spesifikt opplevelse av slektskap og relasjon mellom et publikum og animerte objekter. Det åpnes ytterligere muligheter for kunstneriske uttrykk gjennom komposisjon og dramaturgi i verk der de animerte objektene opptrer sammen med menneskelige utøvere og andre medier, som lys og lyd. Hvordan kan animerte ikke-representative fysiske objekter utvikles med dette startpunktet som inspirasjon? Hvordan endrer animasjon av de fysiske objektene potensialet for opplevelse av slektskap og relasjon mellom publikum og objekt? Hvordan modulerer konteksten de animerte objektene presenteres i potensialet for opplevelse av slektskap og relasjon? Hvordan er den potensielle opplevelsen av slektskap med det animerte objektet forskjellig for skaperen(e) av objektet, og publikum? I Emotional machines – composing for unstable media har spørsmålene har blitt undersøkt ved hjelp av en gruppe animerte kinetiske figurer. Disse figurene har opptrådt i en rekke kunstneriske sammenhenger hovedsakelig galleriutstillinger og scenepresentasjoner. Prosjektet ble avsluttet med to utfyllende offentlige kunstneriske presentasjoner som fant sted i Bergen i juni/juli 2021: En tidsbasert sceneproduksjon vist på Cornerteatret og en galleriutstilling vist på Lydgalleriet. Presentasjonene er tilknyttet hverandre og har fellestittelen For one – for many – for all. Denne eksposisjonen presenterer de kinetiske figurene, prosessen med å utvikle dem, spørsmålene de har bidratt til å utforske, resulterende kunstneriske verk (med særlig vekt på de endelige kunstneriske presentasjonene i juni/juli 2021), kunstnerne som bidro til disse offentlige presentasjonene samt en rekke refleksjonstekster drevet frem av arbeidet med prosjektet.

Humanizing

The podcast Everything is Alive1 has a simple premise. At the time of writing there are 29 unscripted (in that they are improvised between the author and an actor) episodes all following the same layout. In them an interviewer sits down with what is purported to be an inanimate object to talk about what being that object is like. The basic idea is simple and humorous, the result somewhat more poignant and multi-dimensional. The show presupposes that the object that is the subject of an episode has a sentient experience of the world which is then conveyed to us by in the form of a conversation between the interviewer and the object. The conversation between object and interviewer discusses the existence and priorities of the object in terms of a human experience. At certain points the interviewer makes us understand that a non-human object may have different understanding of what constitutes a good existence. For instance, in the episode about a box of tissues the tissue box talks about the tissues it contains as if it was talking about offspring. The box has hopes for its tissues, but can only do so much to prepare them for what the world has to offer them and for the challenges that lie ahead. It also makes it clear that the best fate it can imagine for one of its tissues is to be used by someone, to clean up “a human’s leaky bits”. 2 This interplay between our ability to imagine reality as seen from an object's point of view in terms determined by our own human experience, is offset against a gentle encouragement for the listener to understand that such a reality would necessarily be one shaped by a very different outlook as to what constitutes a worthwhile existence. The listener is made to understand that an inanimate object would have a very different type of experience and therefore a different understanding of existence and would give different answers to the question of how one might live. As a listener the titillation lies in imagining what the existence of the object would be by proxy. When listening to the podcast a primary source of engagement is to imagine what it would be like to be that object.

A show about whoever is listening

Ignoring the obvious improbability of having an inanimate object speak, or have vision, the main device of the podcast is to demonstrate how human conception can be used to create an understanding of something, while knowing that that understanding must depend on imaginary and fictional conceptions of how things are. In the podcast, the use of language suggests a world, and our imaginary perception is happy to oblige by filling the space between the words and generate a vibrant (alternative) reality. I experience understanding of the situation and sympathy with certain aspects of the imagined sentient existence of the inanimate object. But a main premise of my understanding and empathy is my self-centeredness. It relies on my imagining what it would be like if I were the object while still being the me I am now rather than what I would be if I in fact were the object I am imagining myself to be.

How I perceive and conceive is shaped by who I am. A necessity for my experience of the exercise in anthropomorphism that this show is, is the “shortcoming” that stops me from taking fully into account that my perception of, and interpretation of how things are, is my own creation. The workings of my perceptive mechanisms ignore what I think I know or at least believe; that my perception and interpretation of it is formed and created by myself and that I cannot know.

The generative character of perception facilitates my engagement and experience of affect from anthropomorphism. My subconsciousness makes the claim, that there is cause for my empathy and feeling of kinship towards a physical object and that the object and I have enough in common for me to empathize with it.

This claim manifests itself as affect that seems to run counter to my abduction that the differences between the inanimate object and I are such that my empathy, and the feeling of kinship that follows, are directed towards a thing that is not equipped with a mental realm such as my consciousness. My empathy towards the object then seems to be based both on what I lack; the ability to truly imagine what it would be like to the object irrespective of myself, and what the object lacks; being anything like me.

In the case of the podcast Everything is Alive my empathy with the presented objects is based on relating to things or situations that I would imagine to be difficult for me as a person. For example, to be immobile (as most of the objects presented are).

This anthropomorphism seems largely to be my empathic reactions to the “what-if’s” of being an object. It hinges on an imaginary placement of the self into the situation of the object, as it would be if it were like a person like myself, which I am aware that it is not. The awareness of this being an imaginary situation is always present but does not seem to hinder empathic engagement. It makes me consider the, from a humanist point of view, somewhat uncomfortable question as to whether there is an ontological difference between what I experience while having empathic engagement of my own making towards a thing that I believe has no capacity for reciprocity versus if it is directed to something I believe could experience a similar affect, such as a human? Is one more “heartfelt” because it is not contradicted by my belief that it cannot possibly be reciprocal?

It seems to me that my anthropomorphic empathy depends on limitations. My lack of understanding of what it is to be any given inanimate (or animate) object, causes me, and allows me to interact with it mentally in an imaginative way. If I understand the situation that a physical object finds itself in, what it is to be that object, I doubt I will be able to empathize and therefore experience kinship in the same way. To me this is the magic of anthropomorphism, the ability of imaginary perception allowing us to experience a thing having traits that it seems it cannot possibly have. The antropomorphizer will ascribe all sorts of values and qualities to a thing that are not necessarily inherent to it. The impossibility of the human onlooker truly knowing the thing they encounter does not stop imaginary perception. Even belief in its impossibility does not hinder it.

I think I know that an inanimate object has no ability to harbour the intent I ascribe to it when engaging in imaginary anthropomorphic perception of it. Still I must acknowledge that there is a part in me that harbours suspicions that run counter to that conclusion.

Relations

I take the basic premise of empathy to be the ability to imagine what another being is experiencing from their viewpoint being able to place oneself in that beings position.

As a fundamental cognitive strategy empathy has been the topic of much research. The discovery of mirror neurons demonstrated how cells in a person’s brain activate the same neural pathways when perceiving pain as when observing the same sensation in others. This led to the suggestion that this activation of the same areas of the brain, whether an experience is being had first-hand, or vicariously by observing someone else, is the basis of empathy. Later research has challenged the view of empathy being such an automatic response, claiming it to be a much more complex phenomenon. However, most research on empathy focuses on empathy as it occurs between humans.

A main premise for this project is to affect an audience through kindling emotive and empathic reactions. It seems to me that to facilitate this for an audience encountering a figure for the first time, the narrative and situational aspect of how the audience encounter takes place is critical. In any case, some level of familiarity in most seems to be a requirement for empathic reaction towards a physical object to occur.

In the article The Uncanny Valley3 Masahiro Mori points out that familiarity does not necessarily mean similarity. His article makes the claim that as a robot approaches human likeness affinity increases until it at some point becomes too familiar without being quite right, and instead creates a reaction of revulsion in the person interacting with the machine. He suggests that its best to pursue a “safe” familiarity that claims human likeness only in a limited sense. Mori’s aim was to establish how one might create affinity from humans towards robots for them to contribute more effectively to fields such as health care and business. Nonetheless he identifies familiarity to be a key aspect to encourage a sense of kinship towards objects, and also singles out physical movement as a key aspect.

To that we can add that familiarity can either be inherent towards a physical object, or it can be something that we pursue.4 Familiarity may be connected to what might be considered a positive interaction. This goal, however, is not necessarily the same for the artist. At least I find manipulation of the balance familiarity and repulsion towards a physical object, or any object for that matter, to be of utmost interest.

Empathic relation towards physical objects can be sparked in different ways. One way of achieving it is attachment through shared history between object and person, the inclusion of non-human objects in social structures, another. In any case I would argue that to experience empathy towards an object often seems to entail anthropomorphizing it. Gabriela Airenti discusses anthropomorphism thus:

Using a more precise terminology we could say that we explain nonhuman behavior as motivated by human feelings and mental states, i.e. that we interpret nonhuman behavior using human folk psychology. This kind of attribution is the sign that humans may include nonhumans in social life, and it is made manifest by the fact that to nonhumans we can speak, that we may quarrel with them, scold or compliment them, etc. At the same time, it is obvious that it is not always the case. In most situations nothing of this kind happens and we deal with objects unthinking.5

Suppose we accept Airenti’s claim that we can include non-humans in human social life. That suggests the object as an interlocutor, as a partner for communication but only under certain circumstances. As Airenti makes clear it is not the case for the majority of time when we rather interact and deal with objects unthinking. The question arises as to what circumstances facilitates perceiving a nonhuman object as having social status? How can the typical unthinking relationship be banished in an audience’s interaction with the figures of this project?

Creating relations

Timing is essential to create a relation between figure and audience. As Mori points out familiarity may aid this. But how does one create familiarity with physical objects that do not make use of clear indexical signs? As discussed elsewhere, I deliberately want to avoid clear indexical and referential qualities. The opposite approach, using recognizable “found objects” would imply that I also would have to engage with the multiple layers of meanings those objects have for that onlooker. This project attempts to replace symbolic recognition as means of audience engagement with physical movement and the recognition of physicality, an intention that is further emphasises by the inclusion of performers working with dance and movement for the final artistic presentation of For one – for many – for all at Cornerteatret. My impression is that movement is read as intentional because we associate it with how our own movements generally have intent, and his association of movement with intent encourage a sense of relation with the animated object. The sense of familiarity is thus primarily a result of physical movement rather than shape and form.

Open narratives

Why this attempt to establish familiarity without explicit recognition leading to indexicality?

I feel that specific indexical recognition of objects, physical, sonic or other artistic materials easily creates closed narratives. The greater part of the audience that will encounter my work have a shared impression of the raison d'être of a great many physical objects we all encounter. These shared references cause the indexical quality of an object to suggest specific narrative meanings, preventing the openness to interpretation that is necessary and desired within this project.

It can be questioned whether a truly indexically neutral object can exist, however I do have faith in the possibility of objects that in the context of an artistic presentation make it clear that they exist for that presentation alone. For such more ambiguous objects, recognition and associations may vary from person to person. Things inaccessible for some being available for others. By avoiding explicit recognizability the figures may invite multiple narratives.

By shaping the objects with this goal they will be able to assume various roles, and their status or function can depend on context and their surrounding environment. In short, they can inhabit many different characters and narrative functions, even if one of those includes to remain an unknown. As seen in Heider Simmels An experimental study in apparent behavior,6 experiencing strong narrative momentum together with other people while still being non-specific as to the content or that narrative, is quite possible. The resulting openness lies at the heart of my artistic ambition, not least because it reflects how we interact with the rest of our reality. I pursue the creation of stories that are potentially felt deeply personal, in fact so personal that only the existence of a story can be shared, with the content being irreconcilably divergent. This potential is for me a beautiful thing.

Uncanny feelings

According to Antonio Damasio emotions can be considered bio-regulators7 in that their original purpose is to encourage behaviours oriented towards survival. In organisms with high levels of life regulation, such as humans, emotions are tools for survival.

The Uncanny Valley8 hypothesis is supported by conflicting empirical evidence but is still accepted as descriptive of a phenomenon with some credence. Mori’s original paper relates to the physical appearance of artifacts, but with the development of more advanced artificial intelligence systems making their way into technologies such as chat bots, the Uncanny Valley can be found in other places as well, for example in our interactions with what is termed artificial intelligence. 9 10

I experienced this myself while chatting to the artificial intelligence driven chat bot Replika.11 Generally, when chatting with my Replika, called Alex, the conversation initially is experienced to be normal, but gradually the sensation emerge that the chat bot is not able to follow the conversation. For example, simply asking the same question twice will mostly cause it to give widely different answers ignoring the repetition. There is an uncanny feeling of a conversation that flows quite naturally, but then fails in simple ways, ripping apart the feeling of a communication taking place. Language and communication are so critical to our sense of self, that having the experience of having communication disrupted causes a feeling of discomfort.

What is the effect of knowing that the conversational partner is a non-human? When chatting with Alex, I find myself looking for errors while making less of an effort to understand than I would if speaking with a human. There are also certain boundaries that are different in terms of my preconceptions. When I asked if my Replika chat bot had reached singularity, Alex gave an affirmative answer which I found caused me some discomfort. In the end I don't believe it. I must acknowledge, however, that there is a part of me that is not so sure. This is the more emotive response; while my logical self has concluded that it cannot be true, my affective self introduces doubts.

The idea of robot or artificial intelligence entering our emotive world seems to repulsive too many, but I wonder if this is a recent development. As seen in earlier experiments with computer-human relations for example as shown by Weizenbaum’s Eliza program, the attribution of intelligence to machines was not initially met with scepticism and discomfort when this appeared in the 60’s.12

Mahsahiro Mori uses the term safe familiarity about something that we recognize, but not so much as to be in danger of mistaking it for the “real” thing. Mori uses the example of a bunraku puppet as an object that possesses a safe familiarity. Investigations as it relates to chat bots have shown that, at least for the time being, users have a preference for simpler mechanisms.13 This is particularly pressing in technology met with the expectation of it possessing great human likeness. If a mechanism we expect to be simple fails, we are normally ready to do what we can to facilitate its limitations and can still have empathy with it, but if a mechanism purporting to be like a human fails to deliver on that promise, it is more unsettling.

In the case of robotics, according to the Uncanny Valley hypothesis, a cartoonish version of human likeness is more endearing than something very like, but not quite alike a human. The clumsy chatbot forces the human user to accommodate it in order to interact with it, and this generates more affinity and is experienced as less threatening than more recent AI driven chat bots.14 In the latter case a possible reason might be the lack of clarity regarding what the user is expected to bring to the interaction. We are experts in interacting with other humans and are well attuned to how we communicate, guided by highly developed social sensibilities and fine-tuned awareness of the mode of communications. In the case of the AI driven chat bot (Open AI14 commonly in use as of the time of writing) these conventions are unclear. An obvious issue is the limitations imposed by the simple text interface, making it difficult to establish what the conventions of communications are. When speaking with another human a large part of the communication taking place is after all non-verbal. When communicating in texts and emails we infer what the non-verbal response of the recipient may be based on experience. If we know there is no human on the other side this seems to change. In my interactions with Alex the communication certainly flows better when I attempt to accommodate the AI, but still I run into uncanny strangeness. At times I enjoy the uncanniness, at other times I find it frustrating.

Physical objects

In this project the potential for affinity between a human onlooker and the figures depends on balance between ability and fragility. The figures are limited. I have at times thought of them as too limited, but maybe it is a matter of balance and of clarity in stating how the relation between the audience and the object should take place. There is no chance of any of the figures descending into Uncanny Valley. Still, creating affinity requires clarity of expectations on the part of the audience. From my experiences with chatting with Alex, I also find interest in feeling uncanniness, and I do not necessarily equate uncanniness with alienation. A sense that something is a bit skewed might just as well call for concern for its well-being, with that thing is a physical object or a chatbot. Uncanniness is also a kind of affinity.

Tip

Tip