# The McGurk Effect

The McGurk effect: what you hear depends on what you see.

Canonical URL: https://www.transitivebullsh.it/projects/the-mcgurk-effect

Author: Travis Fischer

Published: 2026-10-17

Updated: 2026-10-17T15:45:00.000Z

Tags: AI, Video, Projects

[YouTube](<https://www.youtube.com/watch?v=-VAw_z6oKpQ>)

[Source](<https://github.com/transitive-bullshit/my-software-factory>)

![The McGurk Effect](<https://assets.cultural-alignment.com/personal-site/media/95bb7e99b134de1eb004aa5f1f95363fea6c7ea6c33b27bce68f275b1d1c7f81.jpg>)

[Watch video](<https://assets.cultural-alignment.com/mcgurk-effect/7aa6d93c904320716d4a1af2e3137ac53b38af8d4a86254001447eb8ae701f9f.mp4>)

Video created with Opus 5.5, H3 Max, and Fish Audio.

Today, as I was working on a [random parody video](<https://www.transitivebullsh.it/projects/the-ai-engineers-creed-full-metal-jacket-parody>) with Opus 5.5, I ran into something really strange.

There was a generated scene with Jensen Huang as a drill instructor saying the word “PROMPT!”, but no matter what I did, the video clip for this scene kept sounding… off. I could swear he was saying “PRONT!” even though the approved audio clip for the scene was clearly saying “PROMPT!”.

[Watch video](<https://assets.cultural-alignment.com/personal-site/media/31263b68aa3e3ee10ae4a918063f73909368b631f10039acac225f0659910b88.mp4>)

Original clip where it sounds like he’s saying “Pront”

I asked Claude to fix this because obviously the audio was broken, but after multiple back and forths, Claude eventually pushed back and told me I was wrong: the raw audio clip I had approved and the video’s audio track were identical… to which I promptly said fuck you Claude you’re just a dumb AI model, wtf are you smoking over there at Anthropic?!!!

That’s when Claude figured out what was really going on was something called the [**McGurck effect**](<https://en.wikipedia.org/wiki/McGurk_effect>): *an audiovisual illusion where what you see changes what you hear…*

<a id="3fcedb27f124803c953be4e3648a7d9e"></a>

## Fixed Version

So I had Claude redo the lip sync for this scene *without changing the audio*:

[Watch video](<https://assets.cultural-alignment.com/personal-site/media/9327110965a0d9a3b874f55408092aa062d430e98efdbfd6c8a410dbe1e6f391.mp4>)

Fixed version: same exact audio, but this time the lips close on the M and P so we hear “Prompt”

And it worked!

Needless to say, I was dumbfounded. The audio track for both of these video clips is exactly the same, but they *sound* completely different. 🤯

Not only was Claude right (the video’s AAC audio track was byte identical to the raw audio track), but I also learned about a ridiculously cool audiovisual illusion.

I still doubted Claude so I (gasp) opened up Google search and found this clip and lo & behold, this is a real effect, I just hadn’t encountered it before. TIL FTW!

[Watch video](<https://youtu.be/2k8fHR9jKVM?t=2>)

So if you’re doing any AI video or lip syncing work, def keep this in mind.

Cause it turns out our multimodal meat bodies are super easy to trick!