Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows


Creative Commons License

Acarturk C., Coskun M., EMİL S.

REVISTA SIGNOS, vol.54, no.107, pp.867-892, 2021 (AHCI) identifier identifier

  • Publication Type: Article / Article
  • Volume: 54 Issue: 107
  • Publication Date: 2021
  • Doi Number: 10.4067/s0718-09342021000300867
  • Journal Name: REVISTA SIGNOS
  • Journal Indexes: Arts and Humanities Citation Index (AHCI), Social Sciences Citation Index (SSCI), Scopus, IBZ Online, Periodicals Index Online, MLA - Modern Language Association Database, DIALNET
  • Page Numbers: pp.867-892
  • Keywords: Modality, deixis, arrows, gestures, multimodality, TEXT, REPRESENTATIONS, DISCOURSE, CHILDREN, SPEECH
  • Middle East Technical University Affiliated: Yes

Abstract

In daily communication and instructional settings, arrows and spontaneous gestures usually convey similar spatial information to interlocutors. They can visualize spatial information in 2D or 3D space based on shared conceptual and spatial representations. The present study investigates the relationship between spontaneous gestures and arrows from the perspective of generation and comprehension in instructional settings. We introduce a corpus of arrows in written multimodal text and a corpus of gestures generated during communication through multimodal instructional material. We also report an experimental investigation that used corpus material to study the impact of gestures on comprehension learning. A comparison of the corpora reveals functional similarities and differences between arrows and gestures and their complementary role in multimodal communication. The findings suggest that arrows and gestures may share a common conceptual space during communication, having a significant impact on comprehension and learning.