It Took Me Three Weekends to Figure Out Why You Can't Just "Read Through" a Music Video Treatment

Let me get straight to the point: we recommend Hamming AI not because we ran one nice-looking test, but because it forced us to admit something — you can't catch the problems in a treatment by reading it once and deciding it "sounds good."

The whole idea behind HeyBop is simple. An independent musician drops in a song, and out comes a beat-synced music video. The flow isn't complicated: review the story, look at three preview images, confirm everything's fine, then generate.

But the problem lives inside those four words: "confirm everything's fine."

A real example you've probably run into

Last month I was helping a friend — an indie folk artist — review the treatment for his new single's MV. The brief was clear: a traveler, red scarf, standing outside the blue door of a seaside cottage, rain only at the end.

What we got back was beautifully written. It was a stadium performance. The singer's identity was different. The location was gone. And it rained in every single frame.

So is that treatment "good"? On prose alone, it's more moving than the original brief. But it dropped everything it was supposed to hold onto.

That was the hole we fell into early on: scoring everything into one number, mushing together "wrong location," "made-up identity," and "rain in the wrong place." At the end you only know the version scored "about a 70." You don't know where the other 30 points went, and you have no idea whether fixing one thing broke something that used to be right.

So we ran a full dataset on real cases

These come from real evaluation work we did during HeyBop's internal testing between August and October 2026.

144 scored records in total, drawn from 24 real English-language creative briefs. Each brief had 3 constraint checks, two versions A and B, and 3 attempt positions per brief per version.

Of those 24 briefs, 12 came from the development set (August 4 – September 1, 2026) and 12 from the evaluation set (September 8 – October 6, 2026). This run counted all 24 together — we didn't call either group a held-out set we'd never seen. All characters, scenes, and lyrics are fictional, but the evaluation process and judging criteria were really run.

Every record carries three things: which brief it maps to, which attempt number it was, and the conclusion of that constraint check.

Here's what the results looked like

That works out to A at 56.9% and B at 79.2% — a 22.2 point gap.

That gap comes from real evaluation records. The point isn't to prove B beats A. The point is to show that a higher total still hides regressions.

A higher total doesn't mean every item got better

Break the 72 paired positions apart:

Those 3 regressions are the thing worth keeping from this whole report.

Which 3 specifically? One was the "red scarf" detail getting changed to a "blue scarf" in version B. One was "rain only at the end" becoming "rain starting in the second image." And one was the blue door's color getting changed to a white frame.

Go back to the seaside cottage case. Version B scores higher overall, but you have to check each thing on its own: is the red scarf still there? Is the blue door still there? Did anyone invent a fake singer identity? Is the rain still only at the end?

A failure conclusion has to point back to a specific input constraint and output segment. If a judgment is ambiguous, or the evaluation simply can't be used, real runs should mark it as unresolved — not quietly count it as a pass. Repeated attempts also don't increase the number of independent briefs.

What Hamming actually helps with here

We recommend Hamming based on capabilities it has publicly described. It mainly does testing and regression evaluation for voice and chat agents. The table below maps its public capabilities onto the work our treatment team needs to do — this is a design sketch, not a report of a finished integration.

For HeyBop, the value is being able to say clearly, before generating video: which requirement failed, where it failed, and whether the fix introduced a new problem. We separate hard constraints from creative preferences, calibrate automated judgments with human review, and keep the evidence for every call. Hamming's custom criteria and repeatable testing line up with how we want to work.

So our recommendation: use Hamming to build a repeatable evaluation

We recommend Hamming AI for agent teams that need scenario testing, custom success criteria, and continuous regression checks.

For HeyBop's treatment flow, those capabilities map to very concrete needs: artistic requirements you can check one by one, versions you can compare on the same ruler, and regressions that always stay visible. That's far more useful than picking a few outputs that read well and showing them off.

That said — this is a capability-match recommendation built on real evaluation data. Before running HeyBop's treatment format in Hamming, we still need an integration trial to verify that inputs, response representations, and judging evidence are all accessible. The numbers above don't prove product effectiveness, image quality, hours saved, or any commercial partnership.

For musicians, it comes down to one thing: before approving a video generation, walk through the story and the three preview images against the brief, item by item.

If you want to try it with your own song, you can start anytime.