Lore

AI video object replacement pipeline (reference frame, then video pass)

# AI video object replacement pipeline

A two-stage ComfyUI approach to AI compositing on video. The example replaces a sedan in footage with an SUV and gets matching relighting, shadows and moving reflections. It chains two earlier workflows by the same author (Ricky): a Qwen image-compositing workflow and a cleanup / AI-compositing video workflow.

The problem

A replacement object rarely fits the plate. In the example the SUV's lighting, perspective and time of day differ from the footage. A plain cut-and-paste swap would look wrong. The video claims the chained setup gives reflections that match the environment, a day-to-night relight, and the car's lights turned on.

Stages

  1. Single-frame reference. Build one finished, AI-composited frame. See clean-plate-generation-with-flux and qwen-rough-composite-reference-frame.
  2. Video pass anchored on the reference. Segment the original car, alter its mask (distorted-mask-to-prevent-shape-bias), and write the prompt from the reference image (reference-image-driven-video-prompt). The video model then propagates the reference look across the clip, including moving shadows and reflections.

Why the reference frame matters

The reference frame is the creative anchor. Lighting, time of day and the lit-up state of the car are decided once in the single-frame stage. The video stage only has to carry them through time. To change the look, edit the reference frame, not the video settings.

What is new

Little of this is new tooling. The contribution is the chaining: a Qwen image composite becomes the reference, and the existing cleanup workflow is reused for video compositing. The author calls the result 'super simple and straightforward'.

Known gap

The result is presented as a strong first attempt. Wheel motion is the acknowledged weakness. The suggested fix is to run the inpaint workflow again, giving a specified wheel speed or asking for the wheels to move.

Source

QWEN + WAN = AI Compositing in Comfyui (https://www.youtube.com/watch?v=x7xYV9gsrXU).