RESEARCH
Research interests
-
Language acquisition, development and learning
-
The multimodal integration of gesture and speech prosody
-
Gesture-speech development
-
Multimodal language learning/communication and embodied cognition
-
The development of children’s oral narrative and other linguistic/cognitive abilities
-
Teacher training
-
Implementation science
The bootstrapping (causal) role of non-referential gestures
in children's cognitive and complex linguistic skills
While research in gesture studies and language development has predominantly focused on the role of referential gestures––those that visually represent the properties of a referent and closely relate to the semantic content of speech––the value of non-referential gestures has received less attention. My research aims to provide insights into how such non-referential gestures might bootstrap children's cognitive and more sophisticated linguistic abilities, such as oral narrative skills. Through experimental studies with children, my research seeks to investigate the mechanisms underlying the learning effects of these gestures on children's oral communication across various learning contexts and discourse types. Understanding the relative significance of different co-speech gestures is crucial to unraveling how gestures impact language development and learning, especially in the context of oral linguistic abilities. For instance, to address this question, I conduct gesture-based training studies.
Relevant publications:
Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borràs-Comes, J., & Prieto, P. (2018). Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 172(8), 168–188. doi: 10.1016/j.jecp.2018.02.004
Vilà-Giménez, I., Igualada, A., & Prieto, P. (2019). Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance. Developmental Psychology, 55(2), 250–262. doi: 10.1037/dev0000604
Vilà-Giménez, I., & Prieto, P. (2018). Encouraging children to produce rhythmic beat gestures leads to better narrative discourse performances. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 704–708). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-143
Vilà-Giménez, I., & Prieto, P. (2020). Encouraging kids to beat: Children's beat gesture production boosts their narrative performance. Developmental Science, 23(6), 1–14. doi: 10.1111/desc.12967
The predictive value of referential and non-referential gestures
in children's oral narrative abilities
Research concerning the multimodal communicative repertoire in infancy has indicated that referential gestures serve as both precursors and predictors of children’s onset of linguistic milestones. My research seeks to broaden the scope of investigation to further understand the mechanisms through which gestures––particularly non-referential ones––contribute to developmental trajectories. Using longitudinal data, I investigate how gestures, produced at various developmental stages and within diverse discourse genres, can predict oncoming changes in oral narrative proficiency.
Relevant publications:
Vilà-Giménez, I., Dowling, N., Demir-Lira, Ö. E., Prieto, P., & Goldin-Meadow, S. (2021). The predictive value of non-referential beat gestures: Early use in parent-child interactions predicts narrative abilities at 5 years of age. Child Development, 92(6), 2335–2355. doi: 10.1111/cdev.13583
Vilà-Giménez, I., Demir-Lira, Ö. E., & Prieto, P. (2020). The role of referential iconic and non-referential gestures in children's narrative production: Iconics signal oncoming changes in speech. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
Gesture-speech multimodal development and communication
(1) Pragmatic and temporal integration between prosody, gesture and speech in children's narrative discourse
How do children develop their multimodal (gesture and speech) linguistic strategies? This research explores the gesture–speech temporal alignment patterns in children’s narrative speech and examines the relationship between information structure (IS) and gesture referentiality in children’s narrative speech from a longitudinal perspective and by assessing the potential differences between different gestures. This research has been conducted using the "Audiovisual corpus of Catalan children's narrative discourse development" (CatND), which consists of narrative productions carried out by 83 children at two time points in development (when children were 5-6 years old, and then two years later).
(Vilà-Giménez et al., 2021)
Accessible at OSF.
Relevant publications:
Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (2023). Multimodal development in children's narrative speech: Evidence for tight gesture-speech temporal alignment patterns as early as 5 years old. Journal of Speech, Language, and Hearing Research, 66(3), 888–900. doi: 10.1044/2022_JSLHR-22-00451
Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (2020). The development and temporal integration of co-speech gesture in narrative speech: A longitudinal study. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
Rohrer, P. L., Florit-Pons, J., Vilà-Giménez, I. & Prieto, P. (2022). Children use non-referential gestures in narrative speech to mark discourse elements which update common ground. Frontiers in Psychology, 12:661339. doi: 10.3389/fpsyg.2021.661339
(2) Gesture-speech multimodal communication across child-directed and adult-directed narrative discourse
How do young adults adapt their prosody and gestures when communicating to children? This research investigates how young adults (future early childhood teachers) modify their prosody and gestures when communicating with children, using the "TEACH-TALK: An audiovisual corpus of future teachers’ child- and adult-directed narratives" (TEACH-TALK) audiovisual corpus. By comparing multimodal cues such as gesture and prosody in child-directed speech (CDS) and adult-directed speech (ADS) in Catalan, this research explores how young adults modify their communicative behaviors to suit different audiences, shedding light on the distinct strategies employed in interactions with children vs. adults.
(Vilà-Giménez et al., in prep.)
Accessible soon at OSF.
Multimodal corpus analysis
Annotation of audiovisual corpora: MultiModal MultiDimensional (M3D) labeling scheme
We have developed the MultiModal MultiDimensional (M3D) labeling system that proposes a multidimensional approach to gesture analysis, in that researchers may code for dimensions of the form of the gesture, independent of its relationship with speech prosody, or of its various pragmatic meanings. At the same time, this labeling scheme challenges a simplistic definition of non-referential (beat) gestures as biphasic rhythmic non-meaningful gestures.
(Rohrer et al., 2023)
Accessible at OSF.
Check the M3D training materials here!
Relevant publications:
Prieto, P., Cravotta, A., Kushch, O., Rohrer, P., L., & Vilà-Giménez, I. (2018). Deconstructing beat gestures: a labelling proposal. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 201–205). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-41
Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Esteve-Gibert, N., Ren, A., Shattuck-Hufnagel, S., & Prieto, P. (2020). The MultiModal MultiDimensional (M3D) labelling scheme for the annotation of audiovisual corpora. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
The development of children's pragmatic and prosodic abilities
We created a test that aims to assess pragmatic abilities from a multimodal perspective (e.g., by taking into account prosodic and gestural patterns). The test assessment thus focuses not only on the verbal cues (speech, prosody) but importantly, it also considers multimodal aspects of communication (facial expression, gestures, posture).
Relevant publications:
Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021). Bridging the Gap Between Prosody and Pragmatics: The Acquisition of Pragmatic Prosody in the Preschool Years and Its Relation With Theory of Mind. Frontiers in Psychology, 12:662124. doi: 10.3389/fpsyg.2021.662124
Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2019). A new tool to assess pragmatic prosody in children: evidence from 3- to 4-year-olds. In S. Calhoun, P. Escudero, M. Tabain, & P. Warren (Eds.), Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia 2019. Canberra, Australia: Australasian Speech Science and Technology Association Inc.
Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021, June 30). Retrieved from osf.io/pyc34