Lore
Generative AI rendering with control nets, reference image and prompt
Generative AI rendering turns a rough 3D layout into a finished-looking shot without simulating light.
How it differs from traditional rendering
- Traditional rendering calculates how light bounces off surfaces.
- Generative AI rendering uses a neural network that predicts what the image should look like. It draws on patterns learned from millions of training images. In the author's words: "Instead of mathematically simulating light transport, a neural network predicts what the image should look like based on patterns it learned from millions and millions of training images."
- The 3D scene supplies only extra information. This is depth maps, outlines or pose data, collectively called control nets.
- The model interpolates the style of a reference image over the shot, steered by a prompt.
- This is not ray tracing. The author says he got a community note on X for calling it that.
Inputs
- Control passes exported from the 3D scene. See blender-outline-pass-workbench-freestyle and blender-depth-pass-z-channel.
- A reference image, either a single start frame or several references. See single-vs-multi-reference-style-guidance.
- A natural-language prompt.
Emergent effects
The model also generates secondary effects from the movement in the scene, including smoke, water splashes and explosive debris. None of these were modelled in 3D. Nothing is simulated, so the results can be convincing or off (see ai-rendering-speed-versus-control-tradeoff).
Apply
Export control passes from your 3D scene and supply a reference image and a prompt. Let the video model generate the shot. To change the style, swap the reference image or the prompt. The 3D work does not need redoing.
The model that makes this practical is covered in skyreels-vace-merged-model. A runnable setup is in free-ai-renderer-comfyui-workflow.