Generative models are unpredictable. That is the one sentence to remember before you start making video with AI. Out of five generations of the same shot, two are usable, one is completely off and two are somewhere in between. In the end I pick one and that one goes into the edit. That picking is the work — and it is exactly what I wanted to move out of my head and into the machine.
So I built a skill for Claude Code wired into my own ComfyUI. I hand it a script, or just a single sentence, and it produces a finished video. It generates its own character and environment references, pins identity anchors so the setting does not fall apart between shots, films each shot, looks at the result and re-shoots whatever did not pass. It is now on GitHub for anyone to use.
What happens when you hand it a script
You type the brief into Claude Code. It can be broken down into scenes, or it can be one line like "two women on a bench by a playground, one asks a question, the other delivers the punchline". From that point on it is on its own:
- it breaks the brief into shots and assigns each one a length, a camera move, the action and the start and end state;
- it generates reference images of characters and locations — including a full character sheet from several angles when that is needed;
- it generates the scene keyframe and shoots the shot from it;
- it pulls control frames out of the finished clip, looks at them and scores six things: character identity, environment, continuity with the previous shot, prompt alignment, motion quality and artifacts;
- anything below threshold gets re-shot — but it changes only what failed, not the whole prompt;
- upscales the finished takes to HD, edits them together and tells you where the file is.
The last batch I put it through was eight short scripted clips with dialogue, 16:9 at 1280×720. A single shot takes 14 to 25 minutes to generate depending on length, so a batch that size is an overnight job — and in the morning you have finished files, not a pile of variants to sift through.
The feedback loop is the only thing that matters
I spent a long time wondering whether this could be built at all. And for a long time it produced nonsense. A character's face changed between shots, the environment lived a life of its own, a man walked out of a room to the left and entered the next scene from the right. The cause was always the same: nobody was looking at the output. The model made a shot, the shot got saved, and on we went.
The turning point was adding one step: look at it and decide whether it passed. Not at the end — after every single generation. It sounds trivial, but it is the difference between a pile of random clips and a video that holds together.
The second thing that turned out to be essential is the rule change only what failed. If character identity did not hold, you add references and strengthen the anchor — the prompt stays. If continuity is off, you feed in the last frame of the previous shot. Rewriting everything at once would also change what already worked — so only the one thing that failed gets touched.
What gets lost in that list
A few things it does along the way that decide whether the result looks like a video or like an experiment.
Standalone images, no video required. You do not have to want a film. You describe what you want to see and you get an image — a portrait, a product shot, an illustration, a background, whatever. The skill picks the model itself based on what you are asking for: a photoreal portrait goes to a different one than a stylised illustration. Practically anything you can describe in a sentence — and when you want to change it, you carry on in sentences too.
Upscaling to HD. Shots are generated at low resolution because that is many times faster, and the finished takes are enlarged afterwards. This is not pixel stretching: it runs through SeedVR2 or RTX Video Super Resolution, which reconstruct detail, so what comes out is a sharp 1280×720 picture or better. Want to go higher, you change one number. For scale: upscaling a ten-second shot takes half a minute, generating the same shot takes a quarter of an hour.
A character sheet from a single photo. You give it one portrait and it produces a turnaround — the same person from six angles. Those panels then serve as references for every shot, so the character does not change face when they turn or the light changes. This is the single biggest difference from generating shot by shot.
Changing clothes without losing the face. Need the same character in a different outfit or a different room? You say it in a sentence ("give her a red jacket", "sit him at the table by the window") and the face stays, because the change is made by an editing model with identity locked, not by generating from scratch.
Continuity across cuts. The last frame of one shot goes into the next one as its first frame. Movement then flows across the cut — nobody walks out of a door to the left and enters the next scene from the right.
Lip-sync onto a finished track. When you already have the line recorded — your own voice, a cloned one, or one from a library — the skill locks it into the generation and the video is shot exactly onto it. Not approximately: word for word.
What it runs on
The skill is not tied to a single model — each job goes to whichever one is best at it, and it switches between them on its own:
- MiniMax H3 — video including speech and lip-sync, character turnarounds
- Z-Image Turbo — fast photoreal images, portraits and references
- FLUX.2 Klein — stylisation, illustration and more complex compositions
- Qwen Image Edit — editing an image from a sentence, and turning the same subject to a different angle
- Krea 2 with Identity Edit — changing clothes and scenery while keeping the face
- SeedVR2 and RTX Video Super Resolution — video upscaling, the first for quality, the second for speed
- Higgs Audio — voice cloning and voicing lines when you do not want the native audio
All of it runs on your own machine. Nothing is uploaded anywhere, nothing is billed by the second, and the output carries no watermarks and no provider's terms.
Where it still falls short — audio
I have to be straight here, because this is the one part I am not happy with.
The default path is MiniMax H3's native audio, meaning the dialogue is generated together with the video, lip-sync included. It works surprisingly well, but with Czech it sometimes mangles a word ending. The skill transcribes the result and compares it against the script, so it flags the problem itself and re-shoots the take — which fixes most of it. The rest is a few minutes of work in post, but it is not fully hands-off yet.
The second path is a connection to Higgs Audio. There you can clone your own voice or pull one from a library, and the video is then generated onto a finished track. Voice quality is excellent; intonation and emotion are the tricky part — sadness comes out as fatigue, irony disappears entirely. I am still working on that.
What you need
The skill generates nothing by itself — it is a control layer. Underneath it you need:
- ComfyUI on your own machine, ideally with a 16 GB VRAM card. MiniMax H3 also takes around 50 GB of system RAM while running, which you have to plan for.
- Claude Code, which is where the skill runs.
- PocketComfy — my app, which the skill takes its workflow templates from (and which also lets you drive the whole of ComfyUI from your phone).
- Models from the list above — only the ones you actually intend to use.
If something is missing, the skill checks against the server and tells you. It will not launch a run it already knows will fail.
Getting it running
The skill has its own repository — Promptlab37/VideoDirector. Clone it and copy it into your Claude Code skills folder:
git clone https://github.com/Promptlab37/VideoDirector.git
cp -r VideoDirector ~/.claude/skills/video-director
claudeThe path .claude/skills/video-director/SKILL.md has to exist. Alongside it you need the PocketComfy repository — that is where the skill gets its workflow templates.
After that you just describe what you want filmed. The repository includes a setup guide (ComfyUI address, install root, where projects are stored) and the full documentation of the process — including the rules for Czech dialogue, the shot length table and the repair matrix the skill uses to decide what to re-shoot.
A repository beats a downloadable package in one respect: I keep refining the skill as I hit new problems while shooting, and git pull brings those fixes to you.
Where to get it
github.com/Promptlab37/VideoDirector — MIT licence, documentation in Czech and English, installation guide included.
Help me polish it
I am releasing this under the MIT licence on purpose. The visual side I consider done; the audio side I do not — and that is exactly where more heads beat one. If you shoot something with it, or you work out how to fix intonation in Higgs Audio, let me know. I will happily fold it into the skill.
