Lore

Generative AI rendering with control nets, reference image and prompt

Generative AI rendering turns a rough 3D layout into a finished-looking shot without simulating light.

How it differs from traditional rendering

Inputs

  1. Control passes exported from the 3D scene. See blender-outline-pass-workbench-freestyle and blender-depth-pass-z-channel.
  2. A reference image, either a single start frame or several references. See single-vs-multi-reference-style-guidance.
  3. A natural-language prompt.

Emergent effects

The model also generates secondary effects from the movement in the scene, including smoke, water splashes and explosive debris. None of these were modelled in 3D. Nothing is simulated, so the results can be convincing or off (see ai-rendering-speed-versus-control-tradeoff).

Apply

Export control passes from your 3D scene and supply a reference image and a prompt. Let the video model generate the shot. To change the style, swap the reference image or the prompt. The 3D work does not need redoing.

The model that makes this practical is covered in skyreels-vace-merged-model. A runnable setup is in free-ai-renderer-comfyui-workflow.