RESEARCH

Research interests

  • Language acquisition and development (in typical and atypical populations)

  • Gesture-speech development

  • The multimodal integration of gesture, speech, and prosody

  • Multimodal learning/communication and embodied cognition

  • The development of children’s oral narrative and other linguistic/cognitive abilities

  • Language and cognition

My PhD dissertation, directed by Dr. Pilar Prieto (ICREA-UPF), examined several issues related to the bootstrapping role of non-referential beat gestures —and their concomitant prosodic prominence— in 5- to 6-year-old children’s narrative abilities, focusing on the multimodal integration of gesture, speech and prosody in language development.

The bootstrapping role of

non-referential beat gestures in

children's cognitive and complex linguistic skills

This project examines the bootstrapping (causal) and predictive role of non-referential beat gestures in children's narrative development.  

(1) Gesture-based narrative trainings using non-referential beat gestures

These studies used a between-subjects training with a pretest and immediate posttest design to investigate whether narrative training with observing beat gestures as well as encouraging children to produce these gestures contribute to improving their narrative performance at posttest.

Relevant publications:

Vilà-Giménez, I., Igualada, A., & Prieto, P. (2019). Observing storytellers who use rhythmic beat gestures improves children’s narrative discourse performance. Developmental Psychology, 55(2), 250–262. doi: 10.1037/dev0000604

Vilà-Giménez, I., & Prieto, P. (2018). Encouraging children to produce rhythmic beat gestures leads to better narrative discourse performances. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 704–708). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-143

Vilà-Giménez, I., & Prieto, P. (2020). Encouraging kids to beat: Children's beat gesture production boosts their narrative performance. Developmental Science, 23(6), 1–14. doi: 10.1111/desc.12967

(2) The cognitive effects of non-referential beat gestures on children's narrative discourse comprehension

Llanes-Coromina, J., Vilà-Giménez, I., Kushch, O., Borràs-Comes, J., & Prieto, P. (2018). Beat gestures help preschoolers recall and comprehend discourse information. Journal of Experimental Child Psychology, 172(8), 168–188. doi: 10.1016/j.jecp.2018.02.004

(3) The predictive value of non-referential beat gestures in children's later narrative production

Relevant publications:

Vilà-Giménez, I., Dowling, N., Demir-Lira, Ö. E., Prieto, P., & Goldin-Meadow, S. (in press, 2021). The predictive value of non-referential beat gestures: Early use in parent-child interactions predicts narrative abilities at 5 years of age. Child Development. doi: 10.1111/cdev.13583

We conducted a longitudinal study that analyzes the speech and gestures produced by 45 14- to 58-months-old children in naturalistic interactions with their caregivers. Results showed that the early spontaneous production of non-referential beat gestures, as opposed to the production of referential iconic gestures and non-referential hand flip gestures, predicts later narrative abilities at 5 years of age. We also carried out a very detailed analysis of children's target sentences with co-speech gestures (beat, flip and iconic gestures) following a pragmatic coding scheme.

Vilà-Giménez, I., Demir-Lira, Ö. E., & Prieto, P.  (2020). The role of referential iconic and non-referential beat gestures in children's narrative production: Iconics signal oncoming changes in speech. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

The study aims to assess the predictive value of gestures in children's narrative development by comparing the effects of referential iconic gestures and non-referential beat gestures produced in narrative corpora. Specifically, we investigate whether iconic gestures and/or beat gestures performed by 5- to 6-year-old children while producing narratives can signal changes in children's later narrative performance two years later. Results demonstrated that referential iconics (as opposed to non-referential beat gestures) that accompanied narratives produced by children at 5 and 6 years of age are significant predictors of later narrative scores two years later.

 

Multimodal corpus analysis and

gesture-speech development

article-0-01A48E690000044D-54_634x400_ed

(1) Gesture labeling proposal

We advanced a comprehensive gesture labeling proposal that highlights the independence of the prosodic and semantic properties of different gesture types and at the same time challenges a simplistic definition of beat gestures as biphasic rhythmic non-meaningful gestures.

 

Relevant publications:

Prieto, P., Cravotta, A., Kushch, O., Rohrer, P., L., & Vilà-Giménez, I. (2018). Deconstructing beat gestures: a labelling proposal. In K. Klessa, J. Bachan, A. Wagner, M. Karpiński, & D. Śledziński (Eds.), Proceedings of the 9th International Conference on Speech Prosody (pp. 201–205). Poznań, Poland. doi: 10.21437/SpeechProsody.2018-41

(2) Annotation of audiovisual corpora: MultiModal MultiDimensional (M3D) labeling scheme

We developed a labeling system that proposes a multidimensional approach to gesture analysis, in that researchers may code for dimensions of the form of the gesture, independent of its relationship with speech prosody, or of its various pragmatic meanings.

 

Relevant publications: 

Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Esteve-Gibert, N., Ren, A., Shattuck-Hufnagel, S., & Prieto, P.  (2020). The MultiModal MultiDimensional (M3D) labelling scheme for the annotation of audiovisual corpora. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

The MultiModal MultiDimensional (M3D) labeling manual represents a joint effort between three gesture labs (Language and Cognition Research Group, UOC; Speech Communication Group, MIT; Prosodic Studies Group, UPF) to design a comprehensive guide for the annotation of multimodal speech corpora, particularly focusing on co-speech gestures, their relationship with prosody, and their semantic and pragmatic properties.

Accessible at OSF.

(3) Gesture-speech multimodal development in children's narrative discourse

Relevant publications: 

Gesture-speech temporal integration in discourse

Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P.  (2020). The development and temporal integration of co-speech gesture in narrative speech: A longitudinal study. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.

Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (in prep.). Multimodal development in children's narrative speech: A longitudinal study of referential and temporal aspects of co-speech gesture.

The relationship between gestures and information structure (IS) marking in discourse

Rohrer, P. L., Florit-Pons, J., Vilà-Giménez, I. & Prieto, P. (under review). Non-referential gestures mark new referents in children’s narrative discourse. Frontiers in Psychology.

unbenannt.png

The development of children's pragmatic and prosodic abilities

We created a test that aims to assess pragmatic abilities from a multimodal perspective (e.g., by taking into account prosodic and gestural patterns). The test assessment thus focuses not only on the verbal cues (speech, prosody) but importantly, it also considers multimodal aspects of communication (facial expression, gestures, posture). 

Relevant publications:

 

Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2021). Bridging the Gap Between Prosody and Pragmatics: The Acquisition of Pragmatic Prosody in the Preschool Years and Its Relation With Theory of Mind. Frontiers in Psychology, 12:662124. doi: 10.3389/fpsyg.2021.662124

Pronina, M., Hübscher, I., Vilà-Giménez, I., & Prieto, P. (2019). A new tool to assess pragmatic prosody in children: evidence from 3- to 4-year-olds. In S. Calhoun, P. Escudero, M. Tabain, & P. Warren (Eds.), Proceedings of the 19th International Congress of Phonetic Sciences, Melbourne, Australia 2019. Canberra, Australia: Australasian Speech Science and Technology Association Inc.