This paper examines emerging theories of perception and their relation to metabolic processes and presents the interactive installation Guilty Pleasures, informed by these theoretical principles. The metabolic nature of perception is particularly apparent in the experiences relating to the oral cavity, and this work explores this through an intra-oral electronic interface, combined with other modalities for enacting illusory sensations of eating together with the exploration of the phenomenology of craving and the pica condition.
Background
The nature of phenomenological experience, with all its richness, bodiliness and ineffable qualities, is deeply intertwined with metabolisms. We evolved sensory mechanisms to allow us to find things to eat (to fatten, but not poison us), avoid being eaten and land a suitable partner to produce offspring of ever better eaters and avoiders of being eaten. More recent mathematical frameworks for cognition and perception suggest that these processes are not just aiding us in metabolic activities (or that metabolic activities just aid us in cognition), but the activity of cognition is itself metabolic. These frameworks might also offer us new ways of thinking about designing experiences through multimodal interactions, particularly in our case, where experience and consumption merge at the oral cavity.
The oral cavity
When discussing the oral cavity, Das Oral from Hartmut Böhme is an excellent reference (Böhme & Slominski, 2013), presenting anthropological, cultural, aesthetic, linguistic and psychological dimensions. On Everyones Lips: The Oral Cavity in Art and Culture (Ruhkamp et al., 2020) is an extensive source of artworks that engage with or depict the oral cavity, together with several noteworthy essays. Based on the accompanying exhibition (2020 - 2021) at Kunstmuseum Wolfsburg, the book ties together the varying cultural significancies ranging from sexuality to vampirism, gustation, language and dentistry. These two comprehensive books would suggest that we are a species very much fixated on the oral, but the idea of experiencing some form of digital environment through the mouth appears quite alien. However, there is also considerable precedent for interactive devices for the mouth. Early examples include electropalatography devices for speech therapy (Koepp-Baker, 1938), and there are numerous examples of prototypical devices in HCI research for accessibility aides (Mougharbel et al., 2013), sensory substitution and augmentation (Bach-y-Rita & W. Kercel, 2003), discrete interactive devices (Gallego Cascón et al., 2019) and for therapeutic treatments (Franzke et al., 2023). The high spatial and tactile acuity of the mouth lends itself to digital interfaces (Bach-y-Rita et al. 1998), but such devices are rarely seen. There are several explanations (ergonomic and technological ones), but perhaps the idea of some immersive digital experience by inserting a device into the mouth is simply too strange.
Perception and Metabolism
The nature of perception is deeply tied up in homeostatic and allostatic processes: we feel hungry so we eat, or we preemptively avoid hunger by seeking out things that either taste good, smell good or look like they might taste good. We can experience the nature of these sensations first-hand, and the mechanism enabling these experiences has evolved with us. The limited number of tastes we can perceive on the tongue allows us to sense nutritious food, avoid metabolically wasteful foods (practically anything tasteless) and be revolted by toxins (Breslin, 2013). Much of our behaviour is guided by this sensing of pleasant and unpleasant (i.e. harmful things): visual perception in humans appears to be not only sensitive to seeing what we can eat but also what will eat us (New et al., 2007). This neo-Darwinian explanation of sensing and acting in the environment is a given, but that there should be any first-person experience at all has been a long-standing source of perplexity and discussion in the philosophy of mind (Chalmers, 1995; Nagel, 1980). Cognitive science, psychology and neuroscience have historically avoided the challenges of investigating the true nature of consciousness or have simply promised that future researchers will provide answers. However, there has been heightened interest in recent decades in approaching this hard problem of consciousness with scientific and mathematical rigour in the hope of finding an explanation. Recent mathematical approaches to perception, namely Karl Friston’s Free energy principle, attempt to provide a unified theory of consciousness and position perception as precisely a metabolic process of entropy aversion in action.
The free-energy principle is based on the ideas of unconscious inference from Helmholtz and stipulates that cognition is the constant process of entropy minimization (Friston, 2010). Organisms try to resist the pull towards disorder and act to maintain predictable sensory states: the reduction of surprise through interactions with the environment. An organism must occupy the states it expects to be in, which, against intuition, are statistically unlikely. As Anil Seth puts it, “There are many more ways of being mush than there are of being alive” (Seth, 2021). The principle fits neatly in enactivist frameworks, as it positions all aspects of experience as integrated into the organisms' ecology (Bruineberg et al., 2018). This process can be during the instances of perceiving the world, but the underlying theory can equally be applied to multiple generations of an organism over an evolutionary time scale. The simple proposition has complex implications and infamously impenetrable mathematical formulations, but there are some simpler takeaways from the ideas: cognition shares fundamental characteristics with any biological system, from single cell microbe to a social network: the need to regulate interactions with the environment to resist a natural pull towards disorder (Friston and Stephan 2007).
The free-energy principle is closely tied to the ideas of Bayesian Predictive Coding. Friston explains the connection succinctly: agents resist the pull to disorder by reducing free-energy (surprise stimulus); this involves acting on the environment to avoid surprises by making Bayesian inferences about the environment (Friston et al., 2023). The main concept of Bayesian Predictive Coding is that the brain generates top-down sensory predictions (controlled hallucinations or generative experience) based on priors (i.e. expectations) through a statistical handling of probabilities. The brain then attempts to reduce prediction errors with bottom-up sensory information, which is typically provided by interactions with the environment. The Bayesian aspect of the theory references the statistical approach used, where probability indicates a degree of belief in an event. Bayesian Predictive Coding provides us with a framework we use in this work to design using multiple modalities to create a gustatory experience. In the Bayesian brain, a single instance of subjective experience (i.e. qualia) such as the taste of a strawberry, is not provided by the sensory stimulus of a strawberry taste alone but built from prior experience/expectation, any stimulus that increases the chance of strawberry flavour being predicted and the likelihood of the stimulus being perceived. This is how cross-modal interactions of all senses give rise to the qualia we readily associate with single modalities. This is particularly discernable in the sensory mechanisms of the oral cavity and the perceptual qualities of gustation: consuming identical foods or drinks in different sensory contexts can lead to vastly different experiences (Wang et al., 2019). Similarly, the addition of colouring to foods can vastly alter what we taste (Spence, 2015) and smell (Morrot et al. 2001).
Hearing with the mouth
In the installation we present in this paper, there is particular emphasis on the role of sound in building an illusory eating experience. We can hear an awful lot of things going on in our mouths; sounds that build a significant part of our experiences of gustation, yet it is often neglected in our attention. The idea of perceiving sound through the mouth has been around for some time. There are several reports of Beethoven using a rod held between his teeth and against a piano’s surface to hear the music despite his hearing loss (Ealy, 1994). A similar method of bone conduction has been used by the Iñupiat people to hear the sounds of underwater species via an oar held against the jaw (Bakker 2024). As early as 1916, a self-experiment was explained in The Electrical Experimenter, which made it possible to experience the bone conduction of sound (Gernsback, 1916). By placing a needle between the teeth and then touching a spinning phonograph record, the soundtrack can be heard. The vibrations are conducted through the teeth and jawbones, which then transmit the sound to the middle ear. A similar setup was explored in a piece by Nam June Paik (Paik, 1963). Our bodies have an amazingly solid model for localizing sound sources spatially (Lutfi, 2008). This is also reflected in the transmission through our bones: a bite on the left side of the jaw can be clearly distinguished from a bite on the right side. Sound perception of our oral cavity is a significant part of the eating experience, telling us a great deal about the texture and quality of the food we eat (Zampini & Spence, 2004). Additionally to intra-oral acoustics, ambient noises have a significant effect on our perception of food and drink (Woods et al., 2011). The physical properties of the consumed food are the source of its texture and, consequently, sound characteristics. However, the influence of auditory information on expectation can go beyond attenuating or amplifying textures and result in illusory experiences of eating. This work makes use of these oral sound sources, together with olfactory and visual cues, to influence the development of a controlled hallucination of eating without the entirety of the sensory information we would have available in normal ingestion.
Pica and the Phenomenology of the Craving
Pica describes a disorder of craving and/or eating non-food substances (Rose et al., 2000). However, it's evident that such cravings, at least those that don’t result in pathological behaviour, are quite normal, and there are many cultural practices of eating non-food. And sometimes, non-food things simply look delicious, even though they clearly shouldn’t. Some objects might have all the sensory characteristics to suggest deliciousness, where there is nothing of nutritional value to be found, or too often, the object is highly toxic, as in the meme/phenomena of detergent pods being perceived as forbidden fruit (Sleight-Price et al., 2018). The phenomena of eating detergent pods could also be a case of supernormal stimuli, where exaggerated sensory qualities hijack an evolved behaviour (Barrett, 2010).
The level of food processing also has an influence on our perception of edibility and risk, with raw, unprocessed foods posing greater short-term risks (Coricelli et al., 2022). However, the greater the extent of processing, the harder it is to distinguish between food and other synthetic objects based on their smell, appearance, texture and physical behaviour. Ultra-processed foods like energy gels are visually almost indistinguishable from non-foods like resins and other viscous fluids which may well have a satisfying or craving-inducing appearance.
Pica can be a serious disorder, but the prevalence of desire or cravings for non-foods is clearly quite a normal response to certain types of stimulus. The prevalence of the behavour among small children is noteworthy; most children explore objects and their environment with thier mouths and will swallow non-foods given the chance. That this behavour exists despite the obvious fitness costs of choking and poisoning suggests that the sensory knowledge they gain has value. The installation we present in this exposition explores the topic of pica and asks what knowledge can be gained by experiencing and eating non-foods without any risks to ourselves.
The Installation
The Guilty Pleasures installation creates a dining experience that simulates the raw perceptual qualities of eating inedible foods through technological means. After sitting down in the booth and selecting the eating experience from the menu, an introductory video is played to guide the user to insert a fresh mouthpiece in the exciter and place the device in their mouth. This device allows participants to experience the sound of fictitious meals being eaten through bone conduction, providing an embodied sensation of chewing and swallowing through acoustic cues. These intraoral acoustics create the illusion of gustation with the support of touch, smell (Fig. 4 & 5) and video of the meal being eaten from a first-person perspective (Fig. 2 & 3).
To create a fitting dining environment for the experience, the visitor takes a seat in a single-person booth, like those found in some Ramen restaurants. The user is prompted to choose between two different kinds of dishes, a smooth or a crispy course. Those options were chosen based on their distinct differences in texture and associated oral stimuli to cue an expectation that then can be provided with partly familiar and partly strange sensory feedback. The addition of ambient sound from standard speakers increases the perception of the isolated sound source within the oral cavity. In order to integrate the experience into the environment as much as possible and to assure visitors they will experience a very new form of dining, a futuristic look was created for the booth. For example, a menu with cryptic abstract illustrations of the dishes was engraved in acrylic glass and illuminated (Fig. 6 & 7). The result is an experience that is both disconcerting and alarmingly real yet still quite playful.
The oral cavity is a very sensitive and intimate area of our body. Experiences around the face and in the mouth feel as though they are very close to the perceiving self: imagine, for example, a pain in the leg compared to the pain of equal strength in a tooth. This stimulus around the face and mouth can be experienced as an alarming close-up in what Dennet calls the Cartesian Theatre (Dennett 1993); an illusory sense of a central location of consciousness at the receiving end of all inputs. Most people report this seat of consciousness to be somewhere just behind the eyes (Bertossa et al. 2008). In addition, the evolutionary urge to assess the risk of objects/substances coming into contact with the mouth is present. These factors lead to a disconcerting experience, so care was taken to find a balance by introducing familiarity and playfulness.
In the sound design of the two dishes, attention was paid to a number of characteristics that were determined experimentally in preliminary tests. We found it important to incorporate cues such as swallowing noises and the sounds produced by our mucous membrane, lips and tongue. These are strongly present in the sound of eating food and are therefore needed to be present. Furthermore, care must be taken to replicate the frequency spectrum as accurately as possible. Sounds that are produced in the mouth by eating have a somewhat muffled quality. The higher frequencies must, therefore be adjusted accordingly to avoid a noticeable discrepancy.
The visual design of the dishes plays with the culinary aesthetics of the synthetic materials. Combinations of substances, shapes and colours were chosen that suggest textures and (imagined) tastes that are familiar to us and are intended to trigger corresponding associations and expectations. Odour combinations were also created that are associated with “freshness” or “richness” and assigned to the respective dishes. These were based on very familiar food smells such as citrus, with a mixture of non-food smells which were derived from essential oils.
Playing with confirming and breaking the expected in an unfamiliar situation is the basis of the installation. A situation which, according to the Bayesian Predictive Coding model, evokes a mixture of expectations, which are then validated and falsified in a playful and experimental way through sensory stimuli in flowing transitions.
Technology
At the centre of the experience is the aforementioned device, which is inserted into the mouth. The device itself consists of an exciter, mouthpiece and housing (Fig. 8). For hygienic reasons, mouthpieces are exchangeable through a pressure-fit locking mechanism. The accompanying odour component is realized through an ultrasonic atomizer and fan setup which gently blows the vaporized odours towards the recipient.
The point of synchronization is important. Ambient sound, image, mouth sound and smell must run in a precise sequence in order to create the associations in a targeted manner. For this purpose, a program was written in which the desired temporal composition of the outputs could be set and which subsequently played the corresponding sequence after the choice of dish.
The whole experience was set up with explainer videos at the beginning and end of how to proceed. The tone of these videos is intended to foster trust in the unfamiliar experience but also to be suggestive of a futuristic, unstaffed restaurant.
Observations
Visitors to this installation suddenly experience facsimiles of familiar sensations, which are at times very convincing but also equally often discordant with reality. We know that unfamiliar tastes and particularly mouthfeels, can elicit strong feelings of revoltion, so we could expect the uncannyness of these new sensations to lead to strong negative reactions. Some visitors and early test subjects did report some level of disgust based on their associations with certain foods or textures. The majority, however, found a surprising experience which led to curiosity and wonder. For example, noting that the sensation of swallowing is primarily in our ears is a strange little moment of realisation, but it is mixed with unfamiliar or missing sensations. Particularly, the absence of a sense of volition in this experience of swallowing is unnerving.
It is worth mentioning that we all have different levels of sensitivity to all the modalities in play in this installation. For hearing and vision, hearing aids and eyeglasses have a levelling effect; the differences in smell and taste sensitivities between individuals, however, are equally significant but typically unnoticed. Smell plays a particularly strong role in the work; there is a natural distribution of smell sensitivity compounded by reduced sensitivity due to nasal congestion or a smoking habit. Predictive Coding suggests a mechanism by which experiences, like smell, can be created by other modalities, particularly when the primary smell stimulus is ambiguous or weak. By incorporating diverse sensory-motor modalities in this installation, we increase the chance of a convincing gustatory experience for all individuals.
Conclusion
Guilty Pleasures offers an exploration of perception and metabolism through the lens of the oral cavity. Through a novel multimodal interface, the artwork delves into the complexities of craving and sensory illusion, shedding light on the interplay between expectation and reality. Bayesian Predictive Coding and the Free Energy Principle gave us important clues on how to design an illusory experience, as well as a framework for understanding the nature of the experience itself and the fundamental ties of all perception to metabolic processes. Observations from participants reveal a diverse range of reactions, from curiosity to surprise but also feelings of uncannyness. By combining the omnipresent theme of food, our cravings and the curiosity of the visitors, we have attempted to provide an experiential entry point to the deeply complex nature of consumption and perception.
Acknowledgements
We extend our heartfelt thanks to Tabea Feuz and Noe Arnold for contributing to the original concept of Guilty Pleasures in the context of the interdisciplinary design module: The Computer of the Mouth in 2022 at the Zurich University of the Arts. Special thanks go to Elena De Carlo for contributing to both the original concept and the final installation. We also express our gratitude to Nadine Cocina for capturing the cover image for this article.
Bibliography
Bach-y-Rita, P., Kaczmarek, K. A., Tyler, M. E., & Garcia-Lara, J. (1998). Form perception with a 49-point electrotactile stimulus array on the tongue: a technical note. Journal of Rehabilitation Research and Development, 35(4), 427–430.
Bach-y-Rita, P., & W. Kercel, S. (2003). Sensory substitution and the human–machine interface. Trends in Cognitive Sciences, 7(12), 541–546.
Bakker, K. (2024). The sounds of life: How Digital Technology Is Bringing Us Closer to the Worlds of Animals and Plants. Princeton University Press.
Barrett, D. (2010). Supernormal Stimuli: How Primal Urges Overran Their Evolutionary Purpose. W. W. Norton & Company.
Bertossa, F., Besa, M., Ferrari, R., & Ferri, F. (2008). Point zero: a phenomenological inquiry into the seat of consciousness. Perceptual and Motor Skills, 107(2), 323–335.
Böhme, H., & Slominski, B. (2013). Das Orale: die Mundhöhle in Kulturgeschichte und Zahnmedizin. Fink.
Breslin, P. A. S. (2013). An evolutionary perspective on food and human taste. Current Biology: CB, 23(9), R409–R418.
Bruineberg, J., Kiverstein, J., & Rietveld, E. (2018). The anticipating brain is not a scientist: the free-energy principle from an ecological-enactive perspective. Synthese, 195(6), 2417–2444.
Chalmers, D. J. (1995). Facing up to the problem of consciousness. Journal of Consciousness Studies: Controversies in Science & the Humanities, 2(3), 200–219.
Coricelli, C., Rumiati, R. I., & Rioux, C. (2022). Implicit and explicit safety evaluation of foods: The importance of food processing. Appetite, 175, 106062.
Daniel, L. (2015). How the unique configuration of the human head may enhance flavor perception capabilities: an evolutionary perspective. Frontiers in Integrative Neuroscience, 9. https://doi.org/10.3389/conf.fnint.2015.03.00003
Dennett, D. C. (1993). Precis of consciousness explained. Philosophy and Phenomenological Research, 53(4), 889.
Ealy, G. T. (1994). Of Ear Trumpets and a Resonance Plate: Early Hearing Aids and Beethoven’s Hearing Perception. 19th-Century Music, 17(3), 262–273.
Franzke, L., Gartmann, J., Bachmann, D., Töpper, T., & Franinović, K. (2023). TOFI: Designing Intraoral Computer Interfaces for Gamified Myofunctional Therapy. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews. Neuroscience, 11(2), 127–138.
Friston, K., Da Costa, L., Sajid, N., Heins, C., Ueltzhöffer, K., Pavliotis, G. A., & Parr, T. (2023). The free energy principle made simpler but not too simple. Physics Reports, 1024, 1–29.
Friston, K. J., & Stephan, K. E. (2007). Free-energy and the brain. Synthese, 159(3), 417–458.
Gallego Cascón, P., Matthies, D. J. C., Muthukumarana, S., & Nanayakkara, S. (2019). ChewIt. An Intraoral Interface for Discreet Interactions. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
Gernsback, H. (1916). Hearing Through Your Teeth. THE ELECTRICAL EXPERIMENTER, 3 (9), 498.
Koepp-Baker, H. (1938). An electrical phonokinesigraph and its applications in the study of speech. University of Iowa.
Morrot, G., Brochet, F., & Dubourdieu, D. (2001). The color of odors. Brain and Language, 79(2), 309–320.
Mougharbel, I., El-Hajj, R., Ghamlouch, H., & Monacelli, E. (2013). Comparative study on different adaptation approaches concerning a sip and puff controller for a powered wheelchair. 2013 Science and Information Conference, 597–603.
Nagel, T. (1980). What is it like to be a bat? In The Language and Thought Series. Harvard University Press.
New, J., Cosmides, L., & Tooby, J. (2007). Category-specific attention for animals reflects ancestral priorities, not expertise. Proceedings of the National Academy of Sciences of the United States of America, 104(42), 16598–16603.
Paik, N. J. (1963). Listening to the music through the Mouth at the Exposition of Music. https://prenjpac-en.ggcf.kr/archives/artwork/n129
Prakash, C., Stephens, K. D., Hoffman, D. D., Singh, M., & Fields, C. (2021). Fitness Beats Truth in the Evolution of Perception. Acta Biotheoretica, 69(3), 319–341.
Rose, E. A., Porcerelli, J. H., & Neale, A. V. (2000). Pica: common but commonly missed. The Journal of the American Board of Family Practice / American Board of Family Practice, 13(5), 353–358.
Ruhkamp, U., Böhme, H., Slominski, B., Bredekamp, H., Thurner, K., Stiglegger, M., Müller, J., Hinrichsen, B., Garve, R., Leonhard, K., & Others. (2020). On Everyone’s Lips: The Oral Cavity in Art and Culture. Hatje Cantz.
Seth, A. (2021). Being You: A New Science of Consciousness (The Sunday Times Bestseller). Faber & Faber.
Sleight-Price, C., Ahlstone, D., & Jones, M. W. (2018). Forbidden Foodways: Tide Pods, Ostensive Practice, and Intergenerational Conflict. Contemporary Legend, 8, 86–114.
Spence, C. (2015). On the psychological impact of food colour. Flavour, 4(1), 21.
Wang, Q. J., Mielby, L. A., Junge, J. Y., Bertelsen, A. S., Kidmose, U., Spence, C., & Byrne, D. V. (2019). The Role of Intrinsic and Extrinsic Sensory Factors in Sweetness Perception of Food and Beverages: A Review. Foods (Basel, Switzerland), 8(6). https://doi.org/10.3390/foods8060211
Woods, A. T., Poliakoff, E., Lloyd, D. M., Kuenzel, J., Hodson, R., Gonda, H., Batchelor, J., Dijksterhuis, G. B., & Thomas, A. (2011). Effect of background noise on food perception. Food Quality and Preference, 22(1), 42–47.