Authors: Anna Xambó and Sara Price //
Tangible interfaces bring together the physical and the digital: tangible objects are computationally enhanced, and are typically used to both control and represent digital data. In 2006, Klemmer, Hartmann & Takayama published the conference paper How Bodies Matter: Five Themes for Interaction Design, focusing on the implications for design of tangible interfaces. The themes of the paper are informed by theories about embodiment in the context of computing, a topic also known as embodied interaction. This term was coined by Paul Dourish in 2001 and developed in his, now, classical book ‘Where the Action is?‘. In this blog post, we review two of the themes identified in the Klemmer paper, that are related to physical-digital interaction, and which are relevant to the MIDAS project: 1) metaphorical mappings; and 2) epistemic actions.
Klemmer et al. (2006) discuss the tensions in designing interfaces for tangible computing systems based on manipulation of artefacts, in particular how it is best to represent a task. The authors highlight that mappings tend to be based on familiar tasks and concrete objects (e.g. 1-1 mappings from physical objects to digital objects), but raise concerns about how to map more abstract and symbolic objects: How can we map body actions (body-centered approach) apart from digital-physical data (technology-centered approach)?
Antle (2009) focuses on designing tangible interfaces for children, looking at cognition associated to bodily actions. The author introduces metaphor as a mechanism to build abstract knowledge, which supports learning abstract concepts via bodily experiences. The author exemplifies the linking between bodily experiences and the development of children’s abstract thinking via metaphors related to motion. For example, ‘metaphorical movement’ takes place when thinking about reaching a conceptual goal, the conceptual idea being based on the movement involved when e.g. reaching for a toy: here the physical movement of reaching a toy is used metaphorically for building abstract knowledge (Antle 2009).
Experiments in ubiquitous computing can inform tangible interface design on potential metaphorical mappings. Antle, Corness & Droumeva explore the use of full-body embodied schemata as a source for conceptual metaphors (embodied metaphors) in an empirical study with adults and children. Participants were asked to perform exercises in an interactive audio environment were they could control sound parameters with bodily movement and positioning: for example, fast/slow speed controls tempo, or loud/quiet activity controls volume. This approach seems to improve usability in terms of learning, but results show that mappings can also be more ambiguous compared to non-metaphorical mappings.
For MIDAS, research on metaphorical mappings can inform ways of conceptualising the relationships between body and technology when observing the different research sites where bodily interactions are particularly relevant (e.g. digital performance or embodied learning).
Klemmer et al. (2006) distinguish pragmatic action vs. epistemic action. Both are based on manipulation of objects, but whilst the first refers to directly accomplishing a task by manipulating objects, the second refers to an explorative attitude when manipulating the objects that supports mental work and a better understanding of the context. An example of epistemic action provided by the authors is moving tiles of letters when playing Scrabble, which supports mental work.
Antle, Droumeva & Ha (2006) conducted an empirical study with children comparing three interaction styles (mouse input, tangible input and traditional hands-on) during a Jigsaw puzzle task. The authors found out that using a tangible interface promoted more spatial problem solving in terms of exploration (epistemic actions) and direct placements. Physicality of objects and digital feedback seemed to support this.
Sakr, Jewitt & Price analysed an empirical study with children to examine the semiotic work of the hands during scientific investigations about the behaviour of light on a tabletop tangible user interface. The study reveals that epistemic movements were a category of hand actions used for scientific inquiry – that supported a better perception of the environment via manipulation, in particular to learn more about the properties of the objects.
For MIDAS, research on different types of manipulative action can also inform about the nature of gestures when observing embodied interaction in the different research sites, including interactive exhibits in experience design, digital augmentation of the body in digital performances, or tabletop tangible interfaces in both social sciences and digital arts.
In sum, how to best develop metaphorical mappings based on bodily movements and promote epistemic actions seems to be an important (and not easy) aspect when designing tangible interfaces. Here we have complemented a conceptual perspective proposed by Klemmer et al.’s seminal paper with other scholars’ empirical work on embodied interaction and cognition. We have seen that children’s interactions on tabletop tangible interfaces and in whole body interaction environments can inform about metaphorical mappings and epistemic actions, and ubiquitous computing research can shed light on embodied metaphors when analysing gestural, and bodily-based interaction.
Antle, Alissa N. (2009) Embodied child computer interaction: why embodiment matters. Interactions. 27-30.
Antle, Alissa N., Greg Corness, Milena Droumeva (2009) What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments. Interacting with Computers. 21: 1–2, 66-75.
Antle, Alissa N., Milena Droumeva, and Daniel Ha (2009) Hands on what?: comparing children’s mouse-based and tangible-based interaction. In Proceedings of the 8th International Conference on Interaction Design and Children (IDC ’09). ACM, New York, 80-88.
Klemmer, Scott R., Björn Hartmann, and Leila Takayama (2006) How bodies matter: five themes for interaction design. In Proceedings of the 6th conference on Designing Interactive systems (DIS ’06). ACM, New York, 140-149.
Sakr, Mona, Sara Price, and Carey Jewitt (in press) The semiotic work of the hands in scientific inquiry. Multimodal special issue of Classroom Discourse.