The use of auditory and visual context in speech perception by listeners with normal hearing and listeners with cochlear implants

Matthew B. Winn, Ariane E. Rhone, Monita Chatterjee, William J. Idsardi

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

There is a wide range of acoustic and visual variability across different talkers and different speaking contexts. Listeners with normal hearing (NH) accommodate that variability in ways that facilitate efficient perception, but it is not known whether listeners with cochlear implants (CIs) can do the same. In this study, listeners with NH and listeners with CIs were tested for accommodation to auditory and visual phonetic contexts created by gender-driven speech differences as well as vowel coarticulation and lip rounding in both consonants and vowels. Accommodation was measured as the shifting of perceptual boundaries between /s/ and /∫/ sounds in various contexts, as modeled by mixed-effects logistic regression. Owing to the spectral contrasts thought to underlie these context effects, CI listeners were predicted to perform poorly, but showed considerable success. Listeners with CIs not only showed sensitivity to auditory cues to gender, they were also able to use visual cues to gender (i.e., faces) as a supplement or proxy for information in the acoustic domain, in a pattern that was not observed for listeners with NH. Spectrally-degraded stimuli heard by listeners with NH generally did not elicit strong context effects, underscoring the limitations of noise vocoders and/or the importance of experience with electric hearing. Visual cues for consonant lip rounding and vowel lip rounding were perceived in a manner consistent with coarticulation and were generally used more heavily by listeners with CIs. Results suggest that listeners with CIs are able to accommodate various sources of acoustic variability either by attending to appropriate acoustic cues or by inferring them via the visual signal.

Original languageEnglish (US)
Article numberArticle 824
JournalFrontiers in Psychology
Volume4
Issue numberNOV
DOIs
StatePublished - Dec 12 2013

Fingerprint

Speech Perception
Cochlear Implants
Hearing
Acoustics
Cues
Lip
Phonetics
Proxy
Noise
Logistic Models

Keywords

  • Audio-visual integration
  • Cochlear implants
  • Context effects
  • Multisensory
  • Spectral degradation
  • Speech perception

ASJC Scopus subject areas

  • Psychology(all)

Cite this

The use of auditory and visual context in speech perception by listeners with normal hearing and listeners with cochlear implants. / Winn, Matthew B.; Rhone, Ariane E.; Chatterjee, Monita; Idsardi, William J.

In: Frontiers in Psychology, Vol. 4, No. NOV, Article 824, 12.12.2013.

Research output: Contribution to journalArticle

@article{abb04c1cdde442359105860735955234,
title = "The use of auditory and visual context in speech perception by listeners with normal hearing and listeners with cochlear implants",
abstract = "There is a wide range of acoustic and visual variability across different talkers and different speaking contexts. Listeners with normal hearing (NH) accommodate that variability in ways that facilitate efficient perception, but it is not known whether listeners with cochlear implants (CIs) can do the same. In this study, listeners with NH and listeners with CIs were tested for accommodation to auditory and visual phonetic contexts created by gender-driven speech differences as well as vowel coarticulation and lip rounding in both consonants and vowels. Accommodation was measured as the shifting of perceptual boundaries between /s/ and /∫/ sounds in various contexts, as modeled by mixed-effects logistic regression. Owing to the spectral contrasts thought to underlie these context effects, CI listeners were predicted to perform poorly, but showed considerable success. Listeners with CIs not only showed sensitivity to auditory cues to gender, they were also able to use visual cues to gender (i.e., faces) as a supplement or proxy for information in the acoustic domain, in a pattern that was not observed for listeners with NH. Spectrally-degraded stimuli heard by listeners with NH generally did not elicit strong context effects, underscoring the limitations of noise vocoders and/or the importance of experience with electric hearing. Visual cues for consonant lip rounding and vowel lip rounding were perceived in a manner consistent with coarticulation and were generally used more heavily by listeners with CIs. Results suggest that listeners with CIs are able to accommodate various sources of acoustic variability either by attending to appropriate acoustic cues or by inferring them via the visual signal.",
keywords = "Audio-visual integration, Cochlear implants, Context effects, Multisensory, Spectral degradation, Speech perception",
author = "Winn, {Matthew B.} and Rhone, {Ariane E.} and Monita Chatterjee and Idsardi, {William J.}",
year = "2013",
month = "12",
day = "12",
doi = "10.3389/fpsyg.2013.00824",
language = "English (US)",
volume = "4",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Research Foundation",
number = "NOV",

}

TY - JOUR

T1 - The use of auditory and visual context in speech perception by listeners with normal hearing and listeners with cochlear implants

AU - Winn, Matthew B.

AU - Rhone, Ariane E.

AU - Chatterjee, Monita

AU - Idsardi, William J.

PY - 2013/12/12

Y1 - 2013/12/12

N2 - There is a wide range of acoustic and visual variability across different talkers and different speaking contexts. Listeners with normal hearing (NH) accommodate that variability in ways that facilitate efficient perception, but it is not known whether listeners with cochlear implants (CIs) can do the same. In this study, listeners with NH and listeners with CIs were tested for accommodation to auditory and visual phonetic contexts created by gender-driven speech differences as well as vowel coarticulation and lip rounding in both consonants and vowels. Accommodation was measured as the shifting of perceptual boundaries between /s/ and /∫/ sounds in various contexts, as modeled by mixed-effects logistic regression. Owing to the spectral contrasts thought to underlie these context effects, CI listeners were predicted to perform poorly, but showed considerable success. Listeners with CIs not only showed sensitivity to auditory cues to gender, they were also able to use visual cues to gender (i.e., faces) as a supplement or proxy for information in the acoustic domain, in a pattern that was not observed for listeners with NH. Spectrally-degraded stimuli heard by listeners with NH generally did not elicit strong context effects, underscoring the limitations of noise vocoders and/or the importance of experience with electric hearing. Visual cues for consonant lip rounding and vowel lip rounding were perceived in a manner consistent with coarticulation and were generally used more heavily by listeners with CIs. Results suggest that listeners with CIs are able to accommodate various sources of acoustic variability either by attending to appropriate acoustic cues or by inferring them via the visual signal.

AB - There is a wide range of acoustic and visual variability across different talkers and different speaking contexts. Listeners with normal hearing (NH) accommodate that variability in ways that facilitate efficient perception, but it is not known whether listeners with cochlear implants (CIs) can do the same. In this study, listeners with NH and listeners with CIs were tested for accommodation to auditory and visual phonetic contexts created by gender-driven speech differences as well as vowel coarticulation and lip rounding in both consonants and vowels. Accommodation was measured as the shifting of perceptual boundaries between /s/ and /∫/ sounds in various contexts, as modeled by mixed-effects logistic regression. Owing to the spectral contrasts thought to underlie these context effects, CI listeners were predicted to perform poorly, but showed considerable success. Listeners with CIs not only showed sensitivity to auditory cues to gender, they were also able to use visual cues to gender (i.e., faces) as a supplement or proxy for information in the acoustic domain, in a pattern that was not observed for listeners with NH. Spectrally-degraded stimuli heard by listeners with NH generally did not elicit strong context effects, underscoring the limitations of noise vocoders and/or the importance of experience with electric hearing. Visual cues for consonant lip rounding and vowel lip rounding were perceived in a manner consistent with coarticulation and were generally used more heavily by listeners with CIs. Results suggest that listeners with CIs are able to accommodate various sources of acoustic variability either by attending to appropriate acoustic cues or by inferring them via the visual signal.

KW - Audio-visual integration

KW - Cochlear implants

KW - Context effects

KW - Multisensory

KW - Spectral degradation

KW - Speech perception

UR - http://www.scopus.com/inward/record.url?scp=84889653147&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84889653147&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2013.00824

DO - 10.3389/fpsyg.2013.00824

M3 - Article

VL - 4

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

IS - NOV

M1 - Article 824

ER -