On this page

Watching the music#

I came across a pair of animated scores on X by Classical Melodies: Handel and Vivaldi, with little glowing sprites hopping across the staff and landing on every note exactly as it sounds. You don’t just hear the counterpoint; you watch the voices take turns.

I wanted one for a song I love: Post Malone’s “Candy Paint,” in a solo piano cover by Molotov Cocktail Piano. The piano strips the song down to four voices: a rolling riff, a lead line above it, a high sparkle on top, and a bass that sits out the verses and comes back for the drops.

Watch original video
The original: Post Malone, “Candy Paint” (2017)

My one rule: no generative video or imagery. My last two music videos, Slow It Down and Sometimes I Think Slow, were built from image and video models. This one has none. A purpose-built engine renders every frame from a transcription of the song, so the sync isn’t approximated. It’s the whole point. The audio is the cover, untouched.

Every note gets a performer#

Each voice gets its own performer, a hot pearl of light that travels the score and touches down on every note as it sounds. Notes start as dark smoked glass and flood with candy paint when a performer lands on them. Held notes fill like a gauge, and the performer rides the leading edge of the fill.

The look is the title. Candy paint is a car finish, so the score is pinstriped in gold onto a black-cherry lacquer at night, with metal flake, clearcoat reflections, and shallow focus.

The camera follows a storyboard: close on the riff for the intro, wide on the hooks, hard cuts with a shockwave on the three drops, and a final rise that reveals the whole song at once, under the credits.

How it was made#

Claude Code with Opus 5.5 built all of it: the song analysis, the engine, the choreography, the camera, and the render pipeline. My job was taste and feedback. It took about a day of back-and-forth, and no generation credits at all.

  1. Transcription. Transkun turned the MP3 into 1,893 piano notes, and librosa found the grid: exactly 90 BPM in E major, with bar 0 at 0.702 seconds.
    The transcription, bar by bar: right hand in pink, left hand in blue. The similarity map below it exposes the song’s sections.
  2. Voices. The notes split by register into riff, lead, sparkle, and bass, and then each voice splits into single lines, one per finger.
  3. Concept art from the real renderer. Instead of mood boards from an image model, three early looks were rendered by the engine itself, so whatever I picked was what I’d get.
    The three directions at the same moment of the first hook: A, Candy Lacquer (top); B, Long Exposure (bottom left); and C, Sleeve (bottom right).
  4. Choreography. Every hop borrows from Disney’s twelve principles of animation: anticipation before takeoff, hang time at the top, squash and stretch on landing, and follow-through.
  5. Storyboard. Twelve story beats and 31 camera framings, each anchored to a bar of the song.
    The storyboard, from the intro to the final rise.
  6. Rendering. Headless Chrome renders each frame with three.js and pipes it to FFmpeg with the matching slice of audio. Every frame is a pure function of song time, so any frame can be rendered on its own.

Directions we tried#

  • Three looks. B painted the melody in light trails like cars on a night freeway, and C printed the score as a risograph record sleeve. I didn’t feel the trails, and C was cute, but A was the only one that belonged to this song.
  • Chords. One performer per voice can only visit one note of a chord, and I kept asking why some dots got skipped. We tried one performer per voice, one per line (“twins”), and budding: extra performers split off the voice’s main performer for a chord and merge back after.
    Single, twins, and budding over the same bars. Budding won.
  • Merging. The first buds merged back after every note, which looked twitchy. Now they keep their identity through short gaps and only merge after about a bar. The bass needed its own rule: its buds merge only once its main performer settles on a held note.
    Before: the buds merge after every note. After: they keep their identity through short gaps.
  • The first frame. X shows a video’s first frame as its preview, and ours was black. Of four posters, I liked Macro best, but cutting from it into the song was jarring.
    Four poster directions: flash-forward, frontispiece, the whole song, and macro.

    So the poster became the first shot. The title lingers for a few seconds while the riff performers fly in to their first notes, and the camera follows them into the song without adding a second before the music starts.

    Four ways to fly the riff performers in: wave, hover and dive, rise, and skip. Skip won.

What I learned#

  • Make sync a test, not a vibe. The audio is the master, and a motion test checks that every performer lands on all 1,893 onsets and never jumps between frames. Choreography changes can’t quietly drift off the beat.
  • Render concept art with the real engine. There’s no gap between the pitch and the product.
  • Chords are where the metaphor breaks. One performer per voice reads beautifully on a single line and falls apart on a chord. Budding keeps one character per voice while still hitting every note.
  • Iterate at low resolution. A 540p review of the whole song renders in about five minutes; the 4K60 master takes about 45. I stopped one 4K render halfway and stuck to previews until the final.
  • The first frame is the thumbnail. Make the poster part of the motion instead of a cut away from it.

Tools used#

  • Claude Code (Opus 5.5): all of the code, analysis, choreography, camera direction, and review renders.
  • Transkun: piano transcription from the MP3.
  • librosa: the tempo grid, key, and sections.
  • three.js: the renderer, with bloom, depth of field, and film grain.
  • Playwright + headless Chrome, and FFmpeg: frame-exact rendering and encoding.
  • Blender: on the table from the start, and never needed.

“Candy Paint” by Post Malone. Instrumental cover by Molotov Cocktail Piano. Not affiliated with either.