6. Conclusion
Fast decision-making is often a critical task that underpins performance in computer games (and real life). Players in competitive games are often faced with the need to make numerous, rapid decisions in which the final outcome is decided based on the player’s immediate awareness of the situation. This requirement is also manifest in many business-related decision-making tasks.
In this experiment, we compared the performance of players in a simple visual decision-making task and a multimodal version of the same eight-second task. Players must choose between one of four possible outcomes within the eight seconds before timing out. The longer players wait, the easier the challenge becomes.
We augmented the visual-only task by adding auditory feedback in the form of a sound that slowly increases in amplitude over the eight-second period. When compared with the visual-only task, we found that there is a significant reduction in the number of timeouts experienced by players in the increasing sound display. This reduction does not seem to come at the expense of performance, as players seem to wait longer and make more correct responses in the increasing sound condition.
An interesting result that needs further validation is that players also seem to wait longer when a non-informative constant sound was added to the display. This result was difficult to validate, as considerable player-to-player variation occurred in the task, with average success rates ranging from 21% to 78%. Some players (n=15) performed at least 10% better in the increasing sound display while others (n=14) performed 10% worse with this display. For some (n=19), performance seems relatively unchanged between display modes.
Such variation in performance has previously been reported with multimodal displays (Nesbitt and Barrass 2004; Nesbitt and Hoskens 2008) and categorized as conflicting, complementary, and redundant (McGee, Gray and Brewster 2000; Pao and Lawrence 1998). Where individuals perform worse with multimodal information, the display can be categorized as conflicting; where they perform better, it can be described as complementary; and where there is no change in performance, the display can be described as redundant. This variability in performance is also worth further study to see if it is consistent among individuals across other multimodal tasks, indicating a particular individual preference, or if it might be mitigated by training. Designing a multimodal display requires an understanding of the user's capability, and possible limitations, for processing information. Even simple parameters such as user's age, individual preferences, and sensorimotor skill may impact on performance in a multimodal interface, so providing clear functionality is essential. The salience of input streams can vary according to context, task, user, and time (Turk 2014). Our observations during this study confirmed these various difficulties.
An alternative classification of multimodal interfaces into “exclusive,” “alternative,” “concurrent,” and “synergistic” systems provides a framework for describing fundamental differences in display design (Nigay and Coutaz 1993). Exclusive interfaces design the modalities to be interpreted separately in a sequential fashion. Alternative interfaces also present modalities in sequence but allow for some integrated interpretation of the signals. By contrast, some systems can be described as concurrent, as information is presented in parallel rather than sequentially, although the modes are not designed to be integrated. Arguably, the preferred goals for multimodal systems are synergistic interfaces (Turk 2014), where modes are presented in parallel and fully integrated for a single aspect of the task. There are different levels to consider when analyzing how the modes are integrated (Lalanne, Nigay, Palanque, Robinson, Vanderdonckt and Francois Ladry 2009), for example at a goal, task, semantic or syntactic level.
The interface described in this study can be described as synergistic, where modes are presented in parallel and integrated at the task level of user interaction. A common myth around regarding multimodal interfaces is that enhanced efficiency is the main advantage of multimodal systems (Oviatt 1997). However, the multimodal interface in this case was not intended to increase performance (response time) but to reduce errors in the task.
The literature on multimodal display is distributed across various theoretical and applied domains. Surveys are available on various aspects of multimodal interaction, but the following surveys are recommended for designers who want to consider: related research in the HCI domain (Jaimes and Sebe 2007), the various principles and frameworks of multimodal interfaces (Dumas, Lalanne and Oviatt 2009; Oviatt 1997), guidelines for multimodal display (Reeves et al. 2004), adaptive multimodal displays (Kong, Zhang, Yu and Xia 2011), and more fundamental aspects of cognitive load and interface performance (Wickens 2008; Hollender, Hofmann, Deneke and Schmitz 2010). From an interface designer’s viewpoint, developing and evaluating multimodal systems remains a significant challenge (Turk 2014). As noted in a recent review (Turk 2014), these difficulties have been identified since the earliest work in the field (Nigay and Coutaz 1993), and despite more recent frameworks to described design and evaluation practices (Chang and Bourguet 2008), more work to identify best practice is still required (Turk 2014).