Language acquisition/development and learning
Communication, language and education
The multimodal integration of gesture, speech, and prosody
Multimodal learning/communication and embodied cognition
The development of children’s oral narrative and other linguistic/cognitive abilities
My PhD dissertation, directed by Dr. Pilar Prieto (ICREA-UPF), examined several issues related to the bootstrapping role of non-referential beat gestures —and their concomitant prosodic prominence— in 5- to 6-year-old children’s narrative abilities, focusing on the multimodal integration of gesture, speech and prosody in language development.
PhD thesis committee: Prof. Dr. Katharina Rohlfing (Paderborn University), Dr. Maria Graziano (Lund University) and Dr. Şeyda Özçalişkan (Georgia State University)
The bootstrapping role of
non-referential beat gestures in
children's cognitive and complex linguistic skills
This project examines the bootstrapping (causal) and predictive role of non-referential beat gestures in children's narrative development.
(1) Gesture-based narrative trainings using non-referential beat gestures
These studies used a between-subjects training with a pretest and immediate posttest design to investigate whether narrative training with observing beat gestures as well as encouraging children to produce these gestures contribute to improving their narrative performance at posttest.
Vilà-Giménez, I., Igualada, A., & Prieto, P. (2019). Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance. Developmental Psychology, 55(2), 250–262. doi: 10.1037/dev0000604
Vilà-Giménez, I., & Prieto, P. (2018). Encouraging children to produce rhythmic beat gestures leads to better narrative discourse performances. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 704–708). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-143
Vilà-Giménez, I., & Prieto, P. (2020). Encouraging kids to beat: Children's beat gesture production boosts their narrative performance. Developmental Science, 23(6), 1–14. doi: 10.1111/desc.12967
(2) The cognitive effects of non-referential beat gestures on children's narrative discourse comprehension
Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borràs-Comes, J., & Prieto, P. (2018). Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 172(8), 168–188. doi: 10.1016/j.jecp.2018.02.004
(3) The predictive value of non-referential beat gestures in children's later narrative production
Vilà-Giménez, I., Dowling, N., Demir-Lira, Ö. E., Prieto, P., & Goldin-Meadow, S. (2021). The predictive value of non-referential beat gestures: Early use in parent-child interactions predicts narrative abilities at 5 years of age. Child Development, 92(6), 2335–2355. doi: 10.1111/cdev.13583
We conducted a longitudinal study that analyzes the speech and gestures produced by 45 14- to 58-months-old children in naturalistic interactions with their caregivers. Results showed that the early spontaneous production of non-referential beat gestures, as opposed to the production of referential iconic gestures and non-referential hand flip gestures, predicts later narrative abilities at 5 years of age. We also carried out a very detailed analysis of children's target sentences with co-speech gestures (beat, flip and iconic gestures) following a pragmatic coding scheme.
Vilà-Giménez, I., Demir-Lira, Ö. E., & Prieto, P. (2020). The role of referential iconic and non-referential beat gestures in children's narrative production: Iconics signal oncoming changes in speech. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
The study aims to assess the predictive value of gestures in children's narrative development by comparing the effects of referential iconic gestures and non-referential beat gestures produced in narrative corpora. Specifically, we investigate whether iconic gestures and/or beat gestures performed by 5- to 6-year-old children while producing narratives can signal changes in children's later narrative performance two years later. Results demonstrated that referential iconics (as opposed to non-referential beat gestures) that accompanied narratives produced by children at 5 and 6 years of age are significant predictors of later narrative scores two years later.
Multimodal corpus analysis and
(1) Annotation of audiovisual corpora: MultiModal MultiDimensional (M3D) labeling scheme
We are developing a labeling system that proposes a multidimensional approach to gesture analysis, in that researchers may code for dimensions of the form of the gesture, independent of its relationship with speech prosody, or of its various pragmatic meanings. At the same time, this labeling scheme challenges a simplistic definition of non-referential "beat" gestures as biphasic rhythmic non-meaningful gestures.
Prieto, P., Cravotta, A., Kushch, O., Rohrer, P., L., & Vilà-Giménez, I. (2018). Deconstructing beat gestures: a labelling proposal. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 201–205). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-41
Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Esteve-Gibert, N., Ren, A., Shattuck-Hufnagel, S., & Prieto, P. (2020). The MultiModal MultiDimensional (M3D) labelling scheme for the annotation of audiovisual corpora. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
The MultiModal MultiDimensional (M3D) labeling manual represents a joint effort between three gesture labs (Language and Cognition Research Group, UOC; Speech Communication Group, MIT; Prosodic Studies Group, UPF) to design a comprehensive guide for the annotation of multimodal speech corpora, particularly focusing on co-speech gestures, their relationship with prosody, and their semantic and pragmatic properties.
Accessible at OSF.
(3) Gesture-speech multimodal development in children's narrative discourse
Gesture-speech temporal integration in discourse
Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (2020). The development and temporal integration of co-speech gesture in narrative speech: A longitudinal study. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.
Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (in prep.). Multimodal development in children's narrative speech: A longitudinal study of referential and temporal aspects of co-speech gesture.
The relationship between gestures and information structure (IS) marking in discourse
Rohrer, P. L., Florit-Pons, J., Vilà-Giménez, I. & Prieto, P. (2022). Children use non-referential gestures in narrative speech to mark discourse elements which update common ground. Frontiers in Psychology, 12:661339. doi: 10.3389/fpsyg.2021.661339
The development of children's pragmatic and prosodic abilities
We created a test that aims to assess pragmatic abilities from a multimodal perspective (e.g., by taking into account prosodic and gestural patterns). The test assessment thus focuses not only on the verbal cues (speech, prosody) but importantly, it also considers multimodal aspects of communication (facial expression, gestures, posture).
Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021). Bridging the Gap Between Prosody and Pragmatics: The Acquisition of Pragmatic Prosody in the Preschool Years and Its Relation With Theory of Mind. Frontiers in Psychology, 12:662124. doi: 10.3389/fpsyg.2021.662124
Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2019). A new tool to assess pragmatic prosody in children: evidence from 3- to 4-year-olds. In S. Calhoun, P. Escudero, M. Tabain, & P. Warren (Eds.), Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia 2019. Canberra, Australia: Australasian Speech Science and Technology Association Inc.