On this page

Today, as I was working on a random parody video with Opus 5.5, I ran into something really strange.

There was a generated scene with Jensen Huang as a drill instructor saying the word “PROMPT!”, but no matter what I did, the video clip for that scene kept sounding… off. I could swear he was saying “PRONT!” even though the approved audio clip for that scene was clearly saying “PROMPT!”.

Poor lip sync: the mouth never closes ⇒ it sounds like he’s saying “Pront”

I asked Claude to fix this because obviously the audio was broken, but after multiple back and forths, Claude eventually pushed back and told me I was wrong: the raw audio clip I had approved and the video’s audio track were identical… to which I promptly said fuck you Claude you’re just a dumb AI model, wtf are you smoking over there at Anthropic?!!!

That’s when Claude figured out what was really going on was something called the McGurck effect: an audiovisual illusion where what you see changes what you hear…

Fixed Version#

So I had Claude redo the lip sync for this scene without changing the audio:

Fixed lip sync: same exact audio, but this time the lips close on the M and P so we hear “Prompt”

And it worked!

Needless to say, I was dumbfounded. The audio track for both of these video clips is exactly the same, but they sound completely different. 🤯

Not only was Claude right (the video’s AAC audio track was byte identical to the raw audio track), but I also learned about a ridiculously cool audiovisual illusion.

I still doubted Claude so I (gasp) opened up Google search and found this clip and lo & behold, this is a real effect, I just hadn’t encountered it before. TIL FTW!

Watch original video

So if you’re doing any AI video or lip syncing work, def keep this in mind.

Cause it turns out our multimodal meat bodies are super easy to trick!