Visual and auditory information are key in understanding speech. When we talk to someone, we don’t just listen to what he says, but we also pay attention to how he says it by moving his lips.
The capacity for human speech is based on the integration of visual and auditory information, something that is evidenced by the fact that we can experience a curious illusory phenomenon: the mcgurk effect
It can be said that this peculiar phenomenon occurs when we hear with our eyes, causing what we hear to change depending on what we see. Let’s discover what this interesting visual-auditory effect consists of.
We tend to believe that our senses work independently: when we hear, we only hear; and when we see, we only see Based on this belief, it would be reasonable to think that a visual stimulus is not capable of distorting our way of perceiving sound. However, the reality is that it can, since our perceptual experiences are the product of a complex process of mixing information, the same mixture that gives rise to a peculiar phenomenon: the McGurk effect.
Surely on more than one occasion you have had a conversation in an extremely noisy environment. Maybe it was in a nightclub, on the terrace of a bar on a busy street, or in a high school classroom. When there is loud background noise, it is difficult for us to understand what the person right in front of us is saying and, in order to understand something, we use the old and instinctive trick of watching his mouth while he speaks.
In these cases, visual and auditory information are not analyzed separately, but rather combined The human brain has a region called the superior temporal sulcus, which is specialized in combining both types of information. In the examples we have given, it would be responsible for combining the phonemes our interlocutor pronounces with the movement of his lips.
Because of this ability to combine multimodal information, The superior temporal sulcus is the neurological stage where the illusory McGurk effect occurs which would be nothing more than the result of an error in the decoding of the message when two different sensory modalities interact, causing what we see not to match what we hear.
If we do a quick search on YouTube, we can find more than one video where this phenomenon is presented in a practical way. This link directs us to a good example of this phenomenon:
In this specific case, The person in the video pronounces /ba/ all the time, however, depending on how he moves his lips you can hear either /ba/ or /pa/
This effect can also occur with other combinations of syllables. For example, it can be achieved with the combination /ka/ (visual) and /pa/ (auditory), which gives rise to the perception of /ta/. Another example would be seeing someone making lip movements that correspond to the syllable /ga/ but while the syllable /ba/ is being uttered it will be perceived as /da/.
The way we hear the same sound varies drastically depending on whether we look at the way the person speaking to us moves their lips. This not only affects the perception of simple sets of sounds such as syllables, but it has also been proven to work with complete sentences, although you have probably witnessed it yourself in some of the situations we have mentioned before.
One of the first findings related to the McGurk effect and the interaction between sensory modalities is that having The ability to see how our interlocutor moves their lips considerably improves the volume of what we hear
It has been seen that it gives us the sensation of hearing phonemes up to 15 decibels louder when we have the sender within our visual field. This occurs even when the acoustic conditions are not adverse, such as being in a room with no sound or in a quiet place.
This phenomenon It was first described in 1976 in an article by British cognitive psychologist Harry McGurk together with his colleague John MacDonald titled “Hearing lips and seeing voices.” His original study was going to consist of investigating the imitation patterns of a group of children who were developing speech skills, and the experiment involved presenting them with several videos of people pronouncing different syllables.
However, an error occurred during playback. The technician in charge of editing the video made a mistake and caused the image and sound to not be synchronized, causing the recording to be seen of a person saying something that did not correspond to the sound that was heard.
As the video was played, McGurk and MacDonald heard a third phoneme instead of the lipped and voiced phoneme. It was serendipity that led these two researchers to discover this peculiar auditory illusion.
Its importance in the study of human speech
The discovery of this effect is considered proof that the visual and auditory systems have evolved together to allow, among other things, better speech processing Our visual system helps us discriminate sounds that are difficult to differentiate, an advantage that deaf people carry out when reading lips.
Being able to see how our interlocutor moves their lips increases confidence in the message perceived through the auditory system. That is, if two independent systems point towards the same solution, in this case the same message, that message is trusted more than if we only receive it through one channel.
It is worth mentioning that The McGurk effect does not occur automatically For it to occur, it is necessary that we pay attention to our interlocutor and, when distracting stimuli, both visual and auditory, are incorporated, this illusion is attenuated.
In fact, this proves that the effect is not due to poor reception of visual or auditory information, but rather to an error in the integration of these two sensory modalities.
Another fact that gives strength to the idea that the visual system supports the auditory system is that, when we see a person talking to us but we cannot hear what they are saying to us at all, Not only is our visual cortex activated, but so is the auditory cortex even if we are not listening to anything.
The McGurk effect and brain dysfunctions
It has been seen that brain injuries and suffer from dysfunctions in reading skills in addition to manifesting mental disorders, influences the probability of the McGurk effect occurring.
People who have had a callosotomy manifest the McGurk effect more slowly. It seems that children with specific language disorders present the McGurk effect in a more weakened form compared to those children who do not present language acquisition disorders or reading-writing difficulties.
Laterality also influences seeing that right-handed people are more likely to experience this effect.
The McGurk effect in different languages
Regardless of which language is spoken, its speakers depend, to some degree, on visual information during speech perception. Nevertheless, It has been seen that the intensity of the McGurk effect varies from language to language seeing that in languages ​​such as Spanish, Turkish, Italian, Dutch, English and German, their speakers experience this effect more strongly than speakers of Chinese and Japanese.
That speakers of Asian languages ​​show the McGurk effect less frequently may be due to the cultural practice of avoiding eye contact Added to this, Chinese and Japanese in particular are two languages ​​with very syllabic linguistic structures, usually of the consonant+vowel and consonant+vowel+consonant type, which makes them especially skilled at detecting syllables regardless of how you move your fingers. lips of his interlocutor.
