FrameWeaver

Problem#

Producing an explanatory video end-to-end — researching a topic, writing a script, generating visuals, recording narration, and assembling the final cut — is a genuinely multi-stage pipeline, each stage with a different tool and a different resource profile. Doing all of that by hand is slow; doing it with one monolithic script that assumes unlimited compute doesn't reflect how most people actually have a single GPU to work with, not a cluster.

FrameWeaver is an automated multi-agent pipeline that takes a natural-language topic and produces a finished explanatory video — research, planning, image generation, and voice synthesis, each handled by a specialized agent, coordinated to share one GPU's VRAM budget across the pipeline.

Motivation#

FrameWeaver started primarily as a multi-agent orchestration experiment — the engineering problem of coordinating several specialized agents (research, planning, image generation, voice synthesis, composition) into one reliable pipeline was the main interest, and generating explanatory videos was the application chosen to explore it through.

Architecture#

Four phases, run by main.py:

1. Research — ResearchAgent searches the web (Serper primary, DuckDuckGo fallback, LLM-only as a last resort) and writes data/database/databaseA.json.
2. Planning — PlanningAgent reads the research output and writes data/scripts/manifest.json via the LLM — the single data contract every downstream agent reads from.
3. Parallel phase — ImageAgent (GPU, FLUX.1-schnell) generates images and explicitly unloads its model afterward to free VRAM; TTSAgent (GPU, IndexTTS-2/Qwen3-TTS) then runs voice synthesis; HyperFramesAgent (CPU, LLM-driven) builds the video composition (data/composition/index.html) after scene timing is rebuilt from the actual TTS audio durations.
4. Render — HyperFramesRenderer shells out to npx hyperframes render (headless Chrome), producing the final MP4.

Show source

The pipeline's VRAM schedule is deliberate: phases 1–2 run on llama.cpp CPU-only (0 VRAM used), FLUX.1-schnell takes roughly 8–9GB and unloads immediately after, then TTS takes roughly 2GB now that FLUX has freed its memory, and the final render runs on CPU. The whole pipeline is designed to fit one consumer GPU's VRAM budget, one phase at a time, rather than assuming everything can be resident at once.

The codebase supports both a fully local stack (llama.cpp + HuggingFace-local FLUX + IndexTTS-2) and a cloud/service-backed alternative (DeepSeek via OpenRouter + ComfyUI-hosted models) as configurable options — the local stack is what actually runs today.

Core Features#

• End-to-end pipeline from a natural-language topic to a finished video, with no manual intervention between stages.
• Checkpointing and resume support (CheckpointManager), so a crash mid-pipeline doesn't mean starting over.
• A FastAPI backend with Celery/Redis-backed async job execution, a job status API, and websocket-based live progress.
• A React GUI (job list, job detail, live logs, settings/preferences panels) alongside the CLI.
• Configurable backends for the LLM (llama.cpp or DeepSeek/OpenRouter), image generation (HuggingFace-local FLUX or ComfyUI), and voice synthesis (IndexTTS-2 or Qwen3-TTS) — the pipeline isn't hard-wired to one stack.
• A benchmarking module for capturing pipeline performance data.

Technical Decisions#

The pipeline supports both a local stack and a cloud/service-backed alternative rather than committing to only one. The original idea was to have the option to run everything locally, but a single consumer GPU carries a real risk of running out of VRAM partway through a multi-model pipeline — so the cloud/ComfyUI path was kept in as a fallback rather than removed, even though the local stack (llama.cpp, HuggingFace-local FLUX, IndexTTS-2/Qwen3-TTS, HyperFrames) is what actually runs today.

Engineering Challenges#

Two problems were roughly equally difficult, and both had to be solved for the pipeline to work at all: getting the VRAM handoff between phases reliable (loading and cleanly unloading FLUX before TTS starts, without running out of memory partway through), and coordinating the multi-agent pipeline itself — retries, partial failures, and resuming from a checkpoint rather than restarting a whole run after one stage fails.

Screenshots#

No sample output (video, image, or audio) exists yet — the pipeline's generated artifacts aren't committed to the repository. A sample rendered video or a screenshot of the React job-monitoring GUI would be a natural fit here once available.

Lessons Learned#

The main lesson so far is that resource scheduling has to be a first-class design concern in a pipeline like this, not something bolted on afterward. Which model runs when, and when it needs to release VRAM for the next stage, has to be part of the architecture from the start — retrofitting that onto a pipeline that assumed unlimited memory would have been far harder than designing around a single GPU's budget from day one.

Current Status#

Actively evolving: 104 commits since the repository was created on 2025-11-17, with the most recent push the day before this article was written. No tagged releases exist. Expect the architecture described here to keep changing as the local-vs-cloud configuration options and the agent pipeline itself continue to develop.

Tech Stack#

Python, FastAPI, Celery + Redis for async job execution, PostgreSQL with pgvector for storage, llama.cpp for local LLM inference, FLUX.1-schnell (via HuggingFace Diffusers) for image generation, IndexTTS-2/Qwen3-TTS for voice synthesis, and HyperFrames (headless Chrome) for final video rendering. The frontend is a React 19 GUI.

Related Content#

Browse the rest of my personal projects.