Lore

LLM caption rewrite for render prompts (in-graph local LLM)

After captioning (Florence-2 detailed captioning), the caption goes to a local language model inside the ComfyUI graph. It works the same way as asking ChatGPT or Claude to rewrite text, but it is automated and runs locally. In the video the node is the Searge LLM node (transcribed as 'search LM') with a 7B model.

Why it is needed

The input is a flat, shadowless image (Flat, shadowless, high-contrast base render). Its raw caption would only describe a flat scene, so the diffusion model would be steered towards a flat result.

Instruction pattern

Adapting to other subjects

The instruction is subject-specific. For a different subject, such as a screenshot of a flower vase, edit the instruction to fit it. The rest of the graph is unchanged.

Next step

The rewritten prompt is formatted with presets in sdxl-prompt-styler-presets.

Related

LLM-assisted prompt refinement and restyling covers the manual, chat-based version of this idea. The LLM node may need manual installation from GitHub.

Adapting the rewrite prompt to non-interior subjects

The rewrite prompt in the video was tuned for real-estate interiors. Stefan found it needed editing for the flower test, because the output otherwise described a living room.

When the subject changes, check the rewrite prompt for subject-specific wording before blaming the sampler or ControlNets. If an interior LoRA is loaded (sdxl-lora-for-targeted-improvement), it may also bias a non-interior subject toward interiors.