Multimodal Video Analysis: Exploring Gaze

Author: Mona Sakr //

MODE Project: Exploring historical place with mobile technologies.

MODE Project: Exploring historical place with mobile technologies.

In our research as part of the embodiment strand of Multimodal Methods for Researching Digital Data and Environments (MODE), there is a particular focus on ‘reading’ the body – interpreting physical interaction – and understanding its contribution to meaning-making. For this, we need methods of data collection and analysis that enable us not only to describe what the body is doing, but also to explore and explain how the use of our bodies in interaction contribute to meaning-making. Video data enables observations of the bodily interaction in space and multimodal analysis can be used to understand what part embodied modes of action – such as gaze, gesture, body posture and movement – play in meaning-making.

Multimodal video analysis involves:

  • Creating a multimodal transcription that documents how gaze, gesture, body posture and movement are being used during a particular episode (figure 1);
  • Exploring the contribution of a particular mode to meaning-making by examining it in the wider context of the activity and asking what role it plays in conjunction with other modes being used.
Figure 1. Screenshot of transcription.

Figure 1. Screenshot of transcription.

To demonstrate what this can reveal about how our physical interaction contributes to meaning-making, let’s consider how gaze is used in a video clip of two students talking to each other during a historical exploration activity conducted using mobile technologies. The clip comes from a recent study we conducted looking at 9-10 year old students’ exploration, via a series of iPad location-tagged tasks, of the local Common and its WWII history.

In the clip, two students are waiting to cross the road and are having a conversation about the deep shelters that were underneath the Common during WWII (figure 2). They had learned about the deep shelters through stimuli in the iPad that was ‘tagged’ to a particular location on the Common, which they are now some distance from. As they verbally reflect on the deep shelters and try to visualize them in the context of the current physical environment, their gaze moves back and forth between the photographic image of the deep shelters on the iPad, the physical environment that they have just come from (associated with the deep shelters), the physical environment directly ahead of them, and each other and the filming researcher.

Figure 2. Screenshot from analysis of Clip 1.

Figure 2. Screenshot from analysis of Clip 1.

In psychological approaches, gaze is used as a way of accessing mental activity since it is taken to be indicative of the focus of an individual’s attention (Deubel & Schneider, 1995; Theeuwes et al., 1998). In studies of social interaction, gaze is considered to be an important resource for communicating with others and it has been linked to the ability to theorise about others’ mental states and processes (Charman et al., 2000). In conversational analysis, the sequencing of gaze is considered to be a key element in the unfolding action of co-participants (Goodwin, 1980). Can anything more be learnt about gaze through the multimodal analysis of video data? Can the use of gaze in the video clip described above be explained through these established approaches to gaze?

Our analysis of the clip suggests that it can. The students’ use of gaze is about more than shifting attention or communicating effectively with others. When they begin talking about the experience of sleeping in the deep shelters, their gaze moves from ahead to behind them, to the location they associate with the deep shelters. They repeatedly look back at this location as they attempt to visualize the deep shelters saying, ‘I just can’t picture it in my head’ (figure 3).

The students could have used other modes to draw attention to the experience of the deep shelters and/or the location associated with the deep shelters. They could have just used speech to reflect on the experience of sleeping in the deep shelters, or used deictic gesture and speech to reference the location that they had just come from. Instead, the students fix their gaze upon the site of interest, which requires an almost 180 degree re-orientation of their bodies. Rather than the gaze being simply a marker of attention or social exchange, it seems to be an important part of the visualization process and engagement with the historical event. By fixing their gaze on the location, the students establish or reinforce a physical connection between the past and the present. They imagine past experiences in the setting of the present-day physical environment and this is facilitated through the use of gaze.

Figure 3. Screenshot from analysis of Clip 1 that shows gaze placed behind.

Figure 3. Screenshot from analysis of Clip 1 that shows gaze placed behind.

Multimodal analysis of video data facilitates inquiry into the contribution of the body in meaning-making. Specifically, it enables a focus on gaze, gesture, body posture and movement, and exploration of what each of these modes is for and why it is used. Looking at the mode of gaze in a single video clip demonstrates how multimodal analysis can be used to challenge and extend previous understandings of gaze that focus on attention and social exchange. Our ‘reading’ of gaze in the context of this video clip suggests that gaze can play a key role in learners’ imaginings of the past in the setting of the present physical environment.

References

Charman, T., Baron-Cohen, S., Swettenham, J., Baird, G., Cox, A., & Drew, A. (2000). Testing joint attention, imitation, and play as infancy precursors to language and theory of mind. Cognitive Development, 15(4), 481-498.

Deubel, H., & Schneider, W. X. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision research, 36(12), 1827-1837.

Goodwin, C. (1980). Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn‐Beginning. Sociological Inquiry, 50(3‐4), 272-302.

Theeuwes, J., Kramer, A. F., Hahn, S., & Irwin, D. E. (1998). Our eyes do not always go where we want them to go: Capture of the eyes by new objects. Psychological Science, 9(5), 379-385.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: