RESEARCH

Research interests

  • Language acquisition/development and learning

  • Communication, language and education

  • The multimodal integration of gesture, speech, and prosody

  • Gesture-speech development

  • Multimodal learning/communication and embodied cognition

  • The development of children’s oral narrative and other linguistic/cognitive abilities

  • Teacher training

My PhD dissertation, directed by Dr. Pilar Prieto (ICREA-UPF), examined several issues related to the bootstrapping role of non-referential beat gestures —and their concomitant prosodic prominence— in 5- to 6-year-old children’s narrative abilities, focusing on the multimodal integration of gesture, speech and prosody in language development.

PhD thesis committee: Prof. Dr. Katharina Rohlfing (Paderborn University), Dr. Maria Graziano (Lund University) and Dr. Şeyda Özçalişkan (Georgia State University)  

Extraordinary Doctorate Award, 2022

The bootstrapping role of non-referential beat gestures in children's cognitive and complex linguistic skills

This project examines the bootstrapping (causal) and predictive role of non-referential beat gestures in children's narrative development.  

(1) Gesture-based narrative trainings using non-referential beat gestures

These studies used a between-subjects training with a pretest and immediate posttest design to investigate whether narrative training with observing beat gestures as well as encouraging children to produce these gestures contribute to improving their narrative performance at posttest.

Relevant publications:

Vilà-Giménez, I., Igualada, A., & Prieto, P. (2019). Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance. Developmental Psychology, 55(2), 250–262. doi: 10.1037/dev0000604

Vilà-Giménez, I., & Prieto, P. (2018). Encouraging children to produce rhythmic beat gestures leads to better narrative discourse performances. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 704–708). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-143

Vilà-Giménez, I., & Prieto, P. (2020). Encouraging kids to beat: Children's beat gesture production boosts their narrative performance. Developmental Science, 23(6), 1–14. doi: 10.1111/desc.12967

(2) The cognitive effects of non-referential beat gestures on children's narrative discourse comprehension

Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borràs-Comes, J., & Prieto, P. (2018). Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 172(8), 168–188. doi: 10.1016/j.jecp.2018.02.004

(3) The predictive value of non-referential beat gestures in children's later narrative production

Relevant publications:

Vilà-Giménez, I., Dowling, N., Demir-Lira, Ö. E., Prieto, P., & Goldin-Meadow, S. (2021). The predictive value of non-referential beat gestures: Early use in parent-child interactions predicts narrative abilities at 5 years of age. Child Development, 92(6), 2335–2355. doi: 10.1111/cdev.13583

We conducted a longitudinal study that analyzes the speech and gestures produced by 45 14- to 58-months-old children in naturalistic interactions with their caregivers. Results showed that the early spontaneous production of non-referential beat gestures, as opposed to the production of referential iconic gestures and non-referential hand flip gestures, predicts later narrative abilities at 5 years of age. We also carried out a very detailed analysis of children's target sentences with co-speech gestures (beat, flip and iconic gestures) following a pragmatic coding scheme.

Vilà-Giménez, I., Demir-Lira, Ö. E., & Prieto, P.  (2020). The role of referential iconic and non-referential gestures in children's narrative production: Iconics signal oncoming changes in speech. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

The study aims to assess the predictive value of gestures in children's narrative development by comparing the effects of referential iconic gestures and non-referential gestures produced in narrative corpora. Specifically, we investigate whether referential iconic gestures and/or non-referential gestures performed by 5- to 6-year-old children while producing narratives can signal changes in children's later narrative performance two years later. Results demonstrated that referential iconics (as opposed to non-referential gestures) that accompanied narratives produced by children at 5 and 6 years of age are significant predictors of later narrative scores two years later.

 

Multimodal corpus analysis and gesture-speech development

(1) Annotation of audiovisual corpora: MultiModal MultiDimensional (M3D) labeling scheme

We are developing a labeling system that proposes a multidimensional approach to gesture analysis, in that researchers may code for dimensions of the form of the gesture, independent of its relationship with speech prosody, or of its various pragmatic meanings. At the same time, this labeling scheme challenges a simplistic definition of non-referential "beat" gestures as biphasic rhythmic non-meaningful gestures.

 

Relevant publications: 

Prieto, P., Cravotta, A., Kushch, O., Rohrer, P., L., & Vilà-Giménez, I. (2018). Deconstructing beat gestures: a labelling proposal. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 201–205). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-41

Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Esteve-Gibert, N., Ren, A., Shattuck-Hufnagel, S., & Prieto, P.  (2020). The MultiModal MultiDimensional (M3D) labelling scheme for the annotation of audiovisual corpora. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

The MultiModal MultiDimensional (M3D) labeling manual represents a joint effort between three gesture labs (Language and Cognition Research Group, UOC; Speech Communication Group, MIT; Prosodic Studies Group, UPF) to design a comprehensive guide for the annotation of multimodal speech corpora, particularly focusing on co-speech gestures, their relationship with prosody, and their semantic and pragmatic properties.

Accessible at OSF.

(2) Gesture-speech multimodal development in children's narrative discourse

Relevant publications: 

Gesture-speech temporal integration in discourse

Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P.  (2020). The development and temporal integration of co-speech gesture in narrative speech: A longitudinal study. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (in prep.). Multimodal development in children's narrative speech: A longitudinal study of referential and temporal aspects of co-speech gesture.

The relationship between gestures and information structure (IS) marking in discourse

Rohrer, P. L., Florit-Pons, J., Vilà-Giménez, I. & Prieto, P. (2022). Children use non-referential gestures in narrative speech to mark discourse elements which update common ground. Frontiers in Psychology, 12:661339. doi: 10.3389/fpsyg.2021.661339

The development of children's pragmatic and prosodic abilities

We created a test that aims to assess pragmatic abilities from a multimodal perspective (e.g., by taking into account prosodic and gestural patterns). The test assessment thus focuses not only on the verbal cues (speech, prosody) but importantly, it also considers multimodal aspects of communication (facial expression, gestures, posture). 

Relevant publications:

 

Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021). Bridging the Gap Between Prosody and Pragmatics: The Acquisition of Pragmatic Prosody in the Preschool Years and Its Relation With Theory of Mind. Frontiers in Psychology, 12:662124. doi: 10.3389/fpsyg.2021.662124

Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2019). A new tool to assess pragmatic prosody in children: evidence from 3- to 4-year-olds. In S. Calhoun, P. Escudero, M. Tabain, & P. Warren (Eds.), Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia 2019. Canberra, Australia: Australasian Speech Science and Technology Association Inc. 

unbenannt.png

Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021, June 30). Retrieved from osf.io/pyc34