While the mouse hovers over the textile textures, a soft tree-part chord of sine tones is generated that changes when moving with the mouse from one of the textiles to another. This harmony is derived from the same pitch-system as the impulses and thus creates a homogenous harmonic background.
Furthermore, while ‘scratching’ across the textiles, the visual texture of the textile-image changes and thus forms a path where the scratching occurred. The thickness of this path depends on the above mentioned time interval when the mouse 'hangs'.
Where the textile-texture was removed, an underlying texture is revealed that consists of an old newspaper article. Here, the intention was to stimulate the user to keep 'scrathing' the surface in order to reveal the underlying text and its meaning, thus providing a sense of development and direction in the use of this environment. The underlying article relates to the overall story behind the "Why Frets?" project and has been written by Nicolas Trépanier in collaboration with Marko Ciciliani. For details about this multi-sectional transmedia project, please refer to the project website.
“Why Frets? – TOUCH”
The goal of this project is to evoke sensory impressions that are related to the sense of touch. It is not about reproducing an impression equivalent to touch, but rather an approximation that embraces the specifities and limitations of a conventional mouse interface.
Different haptic impressions are among other result of the surface characteristics of the material that is touched. When swiping over a surface with a finger, a smooth surface offers little resistance to the touching skin, while a rough one tends to lock in with it, which can lead to a stuttering and irregular movement. With “Why Frets? – Touch” I attempted to simulate the latter by making the user move with a pressed mouse across the image of a textile. Depending on characteristics of the image of the textile, the movement of the mouse was impeded, leading to stuttering motions of different intensities. Each step of the stuttering motion also triggered a percussive sound. The pitch, resonant timbre and duration of the percussive sound also depends on characteristics of the textile’s image.
Each time the website is loaded, a different selection of three images of textiles is selected and as a consequence,the 'haptic' behavior is different with each of them.
Direct access to the net-art page: https://whyfretstouch.iem.sh
Conclusion
"WhyFrets - TOUCH" is an exercise to translate the experience of touch to the operation of a simple mouse-interface in an audiovisual environment. Freezing mouse movements are usually encounteredas a malfunction when the computer is overstrained. The accompaniment of each 'stutter' with a sound, plus the underlying sonusoidal harmony should make it clear that no malfunction is at hand (usually, when such malfunctions occur, the entire system freezes for a moment).
The stuttering motion only happens when the mouse is pressed, so even, though it is a minimal effort, it requires the bodily action of pressing, which can be associated with engaging with a 'real' physical surface. The assumption is that the experience of 'scratching' would be weaker if it would occur while moving the mouse even without pressing it.
Based on my personal evaluation, I definitely would not contend that "Why Frets? - TOUCH" offers an equivalent of a real-world haptic experience. In my experience it works, though, as a touch-like translation in the digital realm. Through the combination of sound and the removal of the textile-texture, real-world phenomena have been copied while also being different from the analog experience in obvious ways (scratching a textile in the real-world produces sounds, but not impulses in an organized pitch system. It might alter, or even damage the surface, but it is unlikely that a newspaper article would be revealed underneath). This is aligned with Riva and Waterworth's (2014 : 213) argument that for creating the effect of presence in virtual worlds a certain familiarity has to be established by simulating real-world phenomena. However, on the basis of such a framing of familiarity connectionsand combinations with phenomena become possible that are completely detached from the real world.
Contrary to seeing and hearing, touch can not be experienced from a distance, it is based on the physical encounter with the material. For this reason, the exploration of touch seemed as an important experiment when tackling the question of liveness and presence with digital media. And indeed, I would argue that this experiment has been successful by providing an experience that allows a more intimate involvement than environments where mouse clicks are requiered for merely functional purposes. "Why Frets? - TOUCH" enables a bodily presence in the audiovisual scene of an entirely different nature than the representations of bodies by avatars. In a more sublime manner it forms a quality of liveness that can be subsumed under the category of Sanden's 'Corporeal Liveness'. For Sanden, Corporeal Liveness is at hand when perception or expression are rooted on corporeality (2013 : 39). Even tough his examples span from presences of resonating materials, via the singing and grunting of Glenn Gould in his recordings, to full body performances of live-electronics, the inclusion of the translation of digital touch-like experiences in this category seems plausible.
References:
Sanden, P. (2013). Liveness in Modern Music, Routledge, 2013
Riva, G., & Waterworth, J. A. (2014). "Being present in a virtual world". In M. Grimshaw (Ed.), The Oxford handbook of virtuality (pp. 205–221). New York: Oxford University Press
Visual criteria found in the pixel information of the images are used to shape the stuttering audiovisual mouse-movement. These are:
- Color – hue, saturation and brightness information of the pixel's color are used to control parameters
- Consistencies – with a radius of 20 pixels, each pixel’s color is compared to the color of its neighboring pixels. If they are similar or even identical, it is assumed that the area in the displayed textile is monochromatic and smooth, otherwise irregularities are assumed.
These parameters control the following:
- the time interval how long the mouse 'hangs' before it can move on, thus the stuttering moveement of the mouse.
- the frequency of the percussive impulse that is created with each stutter. It consists of a filtered impulse that is sent through a resonant filter
- the duration of ringing-out of the resonant filter
Once a certain amount of the surface has been revealed (in the code this is controlled by a fixed number of 'stutter'-steps) a transition section is triggered during which the user has no more control over the scene. From a structural point of view, this is comparable to a CUT-SCENE in computer games, but instead of consisting of a preproduced film that is played bay, here the transition is algorithmically generated.
This transition-scene adds circular 'holes' to the textile texture and thus reveals more and more of the undelying newspaper texture. This happens at a high pace and as with each 'hole' also an impulse is triggered, the result is a granular cloud of percussive sounds which raises in frequency. Gradually the density of the texture decreases and the impulses gain in duration. During this process, the newspaper is tinted from its original white color to red, so it starts to blend with the dark red background color of the webpage.
Once the transition reaches the end, only the newspaper is visible. The red taint returns to the original color and the sound dies out. The audiovisual experience has come to an end and the user can now read the article, and scroll to its bottom.