Version 1 · 15 November 2026

The FMF Benchmark: can a model hold a job on a show?

Version 1, 15 November 2026. Built from the making of episode 1 of Friday Morning Funeral, a 21-minute animated sketch show in one felt-puppet style. Every result below is from a job we ran and paid for; a cell we did not run says untested.

What we are looking for

Endurance, above all. One good clip proves little. A show needs the same characters, in the same style, with the same details, across hundreds of shots and several rounds of review. So the headline number here is not how good a model's best clip is. It is how much of a real shot list a model gets through, and how much of that is still in the cut after a person has watched it.

Second: movement and character detail. What kinds of motion a model can add to a drawing without changing who is in it. This is the bar for a premium animated look: living characters whose design never moves.

How a test is set

• Our drawing is the first frame and the last frame. A hold uses the same drawing at both ends (a loop). A move uses drawing A and drawing B. The model fills the time between. A model that cannot take a last frame is tested from the first frame only and marked so.
• Same drawings, same instruction, same day for every model.
• 720p, 16:9, 3 to 5 seconds (the model's own allowed lengths), no sound unless the test is speech.
• Each instruction carries the same rules: no new characters or objects, no text, eyes stay open, every flower stays on its head and in frame.
• No retries inside a score. A second attempt is recorded as a second job.
• A refusal is recorded and not worked round.

The test shots

Each is a shot the pilot needed. A model is run on all of them; the hold and move tests are run on a list of at least twenty set-ups, not on one.

What is measured

Computed on every frame, by the show's own tools:

Endurance, in five levels:

Production yield = E2 × E3. That is the share of an episode's moving shots a model delivers. We call 90% or more holds a show, 70% or more holds with a checker, 50% or more needs a fallback for every shot, and less than that tests only.

Price is dollars per clip that reaches the cut: the price of every job, rejects included, divided by the clips that survive.

The scoring sheet

One sheet per model per run.

Model:                      Version / date:               Run by:
Input accepted:  first frame [ ]  last frame [ ]  reference images [ ]  sound [ ]
Lengths allowed:            Minimum billed:               Price per second:

Test   Jobs  Ends on drawing  Head drift mean/max  Mid-clip  Added things  Identity+mark  Verdict  Note (name the failure)
H1
H2  (list of 20+)
H3
M1  (list of 20+)
M2
M3
A1 / A1 variant
A2
A3
A4
L1
S1
E1 mouth / blink / prop / look left
R1 (each refusal, in the model's own words)

E2 pass rate over the list:        E3 survival after review:        Production yield:
E4 spread over five runs:          E5 clips chained:
Dollars per surviving clip:
Named failures, each with its clip:

Results so far (episode 1, 10 to 15 November 2026)

Video models: holds and moves over a real shot list (H1, H2, M1; levels E2 and E3)

From the log of every hold and pose change of cut 4, and the clips later removed after review.

Thirteen more Kling jobs were refused by the account's limit on jobs in flight, not by the model; they are not scored. Five dropped clips could not be matched to a model. By the yield bands above: Kling 3.0 standard holds with a checker; Veo 3.1 Lite needs a fallback for every shot.

Named failures:

• Veo 3.1 Lite: a character told to hold gestures anyway (most of its mid-clip failures); sparkles, specks and light over dark screens and when hands rise.
• Kling 3.0 standard: its blink on dark bead eyes renders as glowing yellow lids (four host loops rejected on the first try; clean when told to keep the eyes open). Its picture sits further from the source on dark, detailed drawings, which shows when cutting between the clip and the drawing.
• One set of living plates for a desk character was switched off in a later cut because the face shook under a drawn mouth (model not recorded).

Video models: measured side by side (scorecard 003)

One character, same drawings, same day. Head drift in pixels at 1280 wide, mean / max.

None of the three changed the character, lost the flower or added an object in these six clips.

Video models: by test

Also run once each: kwaivgi/kling-v3.0-pro (one clip from one frame: real gestures, felt held, the face drifted mid-clip in a wide frame with a small face; $0.56) and google/gemini-omni-1.2-flash (S1: line exact, lips match; $0.28 for 4 s).

L1 lettering: every video model we animated lettering with garbled it (models not recorded per clip). The show's rule since: lettering is never inside a clip.

R1 refusals: a video model refused a child kicking an adult (model not recorded); the kick is drawings. A voice service refused to design a child's voice.

Seedance 2.0's shots were checked on frame sheets, not with the drift tools. Its row is eleven shots in twelve jobs, too few for a yield figure.

Image models (E1 of the test list)

One lettering failure not attributed to a model: an edit instruction ending "…reads exactly: X. Everything else stays the same" came back with the instruction written into the picture.

Levels not yet run

E1 per-second curves, E4 (five runs of one job) and E5 (chains without anchors) have not been run on any model. H3, M2, M3, A1 to A4 have been run on one or two models each, as production shots. An earlier clip of the car peeling away flickered in the corners and was replaced by a still plate (model not recorded).

What works when animating a character, and what does not

From seven cuts of one episode.

Holds up - Breathing, weight shifts, hair and petals moving, a room alive behind a still character. - One clear movement between two of our drawings, three to five seconds long. - A camera move between two framings of the same set when both framings are our drawings. - Background travel behind figures seen from behind. - A violent, short beat in close-up (a shake, a scream) when it ends on our drawing. - Highlights in dark eyes in a night shot, when asked for.

Breaks - Blinks on dark bead eyes: glowing lids. Tell the model to keep the eyes open; draw the blink. - Raised hands and screens: light, sparks and specks appear around them. - "Hold": cheaper models gesture anyway. - Small faces in wide frames: the face drifts. Faces also stretch when a model opens a mouth or lifts a brow. - A character who is not in the first frame: arrives with the wrong skin or clothes. Generate the exit and play it backwards. - The mark on the head: pushed out of frame by a push-in, or moved, when only a first frame is given. - A rule about the mark ("every flower stays on its head") makes a model add flowers to a shot with no characters in it. - Lettering, always. - Scale between shots: the same prop at two sizes in two clips. - Crowds in a burst of action: the crowd swells. - Talking: lips can match and the result is still wrong for a stylised face. - Anything longer than a few seconds, or without an end frame.

So the premium bar, as a method: drawn keys; a model only between two keys; drawn mouths and blinks on a locked face; every clip measured, vetted for added things and looked at; a still wherever a clip fails.

Running a new model through it

What we need from a lab: API access; first-frame and last-frame input; 720p at 16:9; lengths of 3 to 5 seconds; a no-sound option; the price per second and any minimum billed length; content rules in writing.

What happens: 1. The model is added to the clips file with its price. Nothing else changes: same drawings, same instructions. 2. The whole battery is run in one day beside at least two models from the table above, as controls. 3. Every clip is measured (end frame, head drift, mid-clip wander, added things) and laid out on frame sheets. 4. A person reviews at speed and marks what would survive a cut. 5. E4 and E5 are run on H1 and M1. 6. The sheet is written, with every clip, pass or fail, and each failure named. 7. The lab sees it first. It is published with the next episode only by agreement; a model that passes is used on screen and credited.

What a lab gets that its own tests do not give: a shot list from a show that ships, failures named with the clip, and a yield number that a producer can budget from.

Limits of version 1

• One style (felt puppets with bead eyes) and one show. A cel or anime style needs its own run; in a one-frame test in an anime style, four of six models kept the character.
• Sample sizes are uneven: 122 jobs for one model, one for another.
• The review that sets E3 was one person's notes on seven cuts. The reviewer for the computed checks worked from frame sheets and could not watch at speed.
• Prices are what we paid through one reseller in November 2026.