Lore

Writing the video prompt from the reference image

# Writing the video prompt from the reference image

In AI video object replacement pipeline (reference frame, then video pass) the video prompt is written from the reference image, not from the source footage.

Method

A vision-language model is given the reference frame (Rough composite as input to a Qwen image model) and asked to describe it as if it were a 5-second video. In the example: the SUV driving at night on the streets of Tokyo.

Why

Result

With the reference image, the prompt and a distorted mask, the video should work 'straight out of the box, just like your reference image', with moving shadows and reflections.

Two-frame variant: chatbot writes the Kling motion prompt

In the Kling 3.0 start/end frame workflow, the motion prompt is delegated to a general chatbot. Upload both the start and end frames to a regular ChatGPT session, say the clip is for Kling 3.0, and describe the desired animation. The chatbot writes the prompt from what the frames show, so the author does not hand-write it. The frame differences are what the prompt describes (Frame difference drives the choreography). Pipeline context: Kling 3.0 product-ad pipeline: product shot to finished clip.