Adaptive multimodal natural language generation

Speech, prosody & corpus-based methods

Multimodality, cognition and evaluation



Multimodality, cognition and evaluation

The aim of this subproject is to gain insights in how users interpret and evaluate multimodal output, and whether or not users prefer particular combinations of modalities to other combinations or prefer unimodal variants. This knowledge can be used to make more informed decisions for automatic multimedia generation. So far, we have carried out several studies to investigate multimodal output and modality effect, focusing on the question how people process and evaluate unimodal and multimodal information in the medical domain.

Van Hooijdonk and Krahmer (to appear) report on experiments that have been conducted to investigate which combinations of modalities (e.g., text vs. picture vs. film clip; text and picture vs. text and film clip; speech and picture vs. speech and film clip) are most efficient and effective for illustrating RSI exercises, paying special attention to cognitive load of learners and the special characteristics of procedural learning (i.e., acquiring skills rather than knowledge).

Another study focused on text-structure and spatial cognition. For this study, we conducted an exploratory thinking aloud experiment in which users had to find information in a large medical web site. The thinking aloud protocols that were collected in this study were analysed in order to gain insights in users’ spatial orientation and search strategies in this domain. In particular, we investigated the way in which spatial descriptions are used to conceptualize users' search tasks. See Van Hooijdonk et al. (2004, 2005, 2006).

Recently we have carried out a study investigating which (combinations of) modalities people choose to answer different types of questions when simulating a QA system. To this end  we conducted two experiments, one in which we asked people to create multimodal answer presentations for different medical questions, and one in which these multimodal presentations were evaluated by other people (van Hooijdonk et al., 2007).

For an overview of all results in this subproject, see Van Hooijdonk (2008).