Lore

ComfyUI

Dive Deeper into AI Art Creation: A ComfyUI Tutorial on Customization and Control

The video argues that refined, creative AI art comes not from one-click generation but from a deliberate creative process: hand-compositing a reference image in photo-editing software from scraped style/asset sources, then feeding that composite into a ComfyUI workflow that chains IP-Adapter style transfer with ControlNet to retain composition while imposing style, with the manual preparation and iterative re-rendering being the actual artistic work.

Olivio Sarikas · 2024-04-20 · English

Key ideas

  1. The video opens by contrasting generic, 'boring' AI art with a set of more refined, controlled results (collage styles, painting styles, abstract crops) it says were made with the workflow being taught.

  2. The presenter states this workflow is done in ComfyUI because trying to replicate it in Automatic1111 'didn't work in the same way,' and ComfyUI gives full control over the process and where elements are placed.

  3. The video pre-empts complaints about ComfyUI's complexity by asserting a definition of art: 'art is the search for a solution for a complex problem that is your own expression,' and that pressing one button is not that process.

  4. Accuracy of the face/subject between input and output is explicitly not the goal; the goal is turning an input vision/idea into a creative output by combining a concept and a chosen style into a final work.

  5. Style/reference sourcing is done outside ComfyUI: Pinterest for style inspiration, Pexels and Unsplash for free (including commercial-use) images, and Envato Elements for stock assets including 3D models that can be posed and downloaded as background-removed renders for photobashing.

  6. The composite reference image is then assembled in a photo editor — Affinity Photo (one-time purchase) or Photoshop/Lightroom (~$10/month) — using techniques like selecting and color-adjusting elements (e.g., a butterfly), reshaping elements to be closer to what the AI model can understand (butterfly wings turned into cat-ear shapes), adding abstract color shapes with blend modes (e.g., Vivid Light), and brush-painting brightness contrast between subject and background.

  7. The core ComfyUI process is described as 'just IP adapter style transfer,' built on the Juggernaut Version 9 Lightning checkpoint, with the positive/negative prompt textbox converted to an input and repositioned below the image so the presenter can render without scrolling and 'stay in the flow.'

  8. KSampler settings for the lightning model are specified: 6 steps, CFG 1.5, sampler dpm++ sde, scheduler normal, denoise 0.9, with the composite image scaled down (to 1000x1334) and VAE-encoded (using the model's VAE) before being used as the latent input.

  9. ControlNet is used with the SaiCanny model for SDXL and its Canny preprocessor at resolution 1024, fed the same composited image so the output stays close to that composition; positive/negative prompts also feed into the ControlNet node.

  10. The first IP Adapter node uses IP-Adapter Plus (high strength) via the IP Adapter Unified Loader, with weight (style transfer, SDXL only) set to 1.25 and start/end set to 0/1 so it applies across the whole process; the presenter encourages experimenting with these values.

  11. A second IP Adapter (with its own loader) can be chained after the first, feeding the model output back through a second Unified Loader and IP Adapter node so two styles are applied 'in series'; by default it's shown bypassed (right-click > bypass turns nodes violet/inactive, and the same toggle reactivates them).

  12. The prompt must describe the intended end result (e.g., 'artistic illustration portrait of a black African woman colorful abstract face paint neon light glitter butterfly wings on her head dark background bright face'); the video notes that even when a style input doesn't visibly change the output much, prompt wording still has an impact on the result.

  13. The presenter demonstrates re-using already AI-generated images as new inputs with changed IP Adapter weight (e.g., 0.75) and denoise (e.g., 0.8, then 0.6) to blend two prior results; denoise 0.6 is shown producing 'a good mix' between a painterly/abstract look and a darker/photographic look.

  14. A further technique shown: cropping out different parts of an original image, resizing them, applying a hue shift, and re-entering that as a new input to get abstract, painting-like outputs.

  15. The video closes by explicitly inviting viewers to 'go really deep into the rabbit hole' and share their own creative methods in the comments.

  16. ComfyUI — A node-based interface for Stable Diffusion image generation that the video uses because it gives 'full control over the process and full control over where the elements are,' unlike Automatic1111 where the presenter says the same result 'didn't work in the same way.'. Apply: Build the described multi-node pipeline (image/style inputs, prompt, ControlNet, IP Adapters, KSampler) inside ComfyUI rather than a simpler one-click UI when precise control over composition and style is required.

  17. IP Adapter (style transfer) — The core node used in the workflow to transfer the visual style of a reference image onto the generation; the presenter calls the whole process 'just IP adapter style transfer.'. Apply: Feed a chosen style-reference image into an IP Adapter node with a high weight (e.g., 1.25) and start/end range of 0 to 1 so the style is applied across the entire denoising process.

  18. IP Adapter Unified Loader — The loader node that supplies the IP Adapter with its underlying model, set here to the 'plus high strength' variant. Apply: Attach an IP Adapter Unified Loader configured to 'plus high strength' ahead of each IP Adapter node in the chain; when chaining a second style, route the model output through a second Unified Loader before the second IP Adapter.

  19. ControlNet (Canny) — A conditioning network that constrains the output to follow the structure/edges of an input image, used here with the SaiCanny model for SDXL and its Canny preprocessor at resolution 1024. Apply: Feed the same composited reference image and the positive/negative prompts into the ControlNet node with the Canny preprocessor so the generated output preserves the composition created in the photo-editing stage.

  20. VAE Encode — The node that encodes an input image into the latent space used as the starting latent for the KSampler, using the checkpoint's own VAE. Apply: Scale the composited image down to a moderate resolution (the video uses 1000x1334) before VAE-encoding it, to avoid feeding an overly high-resolution image into the sampler.

  21. Lightning-model KSampler settings — A specific low-step sampler configuration tuned for SDXL-Lightning checkpoints, given as 6 steps, CFG 1.5, sampler dpm++ sde, scheduler normal, and denoise 0.9. Apply: When using a lightning/turbo-style checkpoint, set the KSampler to these values as a starting point, then adjust denoise per-image to control how strongly the input latent is retained versus regenerated.

  22. Juggernaut Version 9 Lightning (checkpoint) — The named SDXL checkpoint model used as the base generation model throughout the demonstrated workflow. Apply: Load this checkpoint as the base model feeding both the ControlNet and IP Adapter chain when replicating the shown pipeline.

  23. Convert widget to input — A ComfyUI right-click function that turns a node's inline widget (e.g., a prompt textbox) into an externally connectable input. Apply: Right-click a text/value widget and choose 'convert to input' to relocate it elsewhere in the graph (the presenter moves the prompt box below the output image to avoid scrolling while working).

  24. Node bypass — A ComfyUI right-click toggle that deactivates a node (shown turning it violet) without deleting it, letting it be skipped or re-enabled on demand. Apply: Right-click an optional node (e.g., a second IP Adapter or second style image input) and select bypass to disable it, then repeat the same action to reactivate it when a second style or image is wanted.

  25. Chained IP Adapters (multi-style blending) — Running two IP Adapter + Unified Loader pairs in series, where the first adapter's model output feeds into the second, so two style references are applied 'working in series' within one generation. Apply: After the first IP Adapter node, route its model output into a second IP Adapter Unified Loader and IP Adapter node loaded with a second style image, then send that combined output into the KSampler as the model.

  26. Photobashing with stock 3D model renders — A technique using Envato Elements' 3D model feature, which lets a user pose a model and download a high-resolution, background-removed render of a chosen angle (not a downloadable 3D file itself). Apply: Pose the 3D asset to the desired angle, click 'download this angle' to get a background-removed render, then composite that render into the reference image being built in the photo editor, keeping in mind lighting and camera angle/distance can't be adjusted.

  27. Blend-mode compositing (e.g., Vivid Light) — A photo-editing technique for layering abstract shapes (e.g., a bright blue stripe) onto an image so the shape reacts to the underlying colors instead of flatly overpainting them. Apply: Add an abstract shape layer over the image and set its blend mode to something like Vivid Light so it interacts with the background tones, then brush-paint brightness adjustments around the subject to control contrast against the background before this composite is used as the IP Adapter/ControlNet input.

  28. Denoise-value tuning for image-to-image blending — Adjusting the KSampler's denoise parameter to control how much a re-used AI-generated input image is preserved versus overwritten in a subsequent render. Apply: Re-run a prior generated image as a new input and step the denoise value (the video tries 0.8 then 0.6) to find a blend point between two source styles, noting that 0.6 produced 'a good mix' in the shown example.

  29. Iterative re-input looping — Feeding an already AI-generated output (or a further-edited crop/hue-shift of one) back into the same pipeline as a fresh input to produce further, more abstract variations. Apply: Take a generated result, optionally crop/resize/hue-shift parts of it in the photo editor, then reload it as the IP Adapter/ControlNet image input with new style/weight/denoise settings to keep iterating deeper into variation ('the rabbit hole').

Insights

The video frames the manual, non-AI steps (sourcing images from Pinterest/Pexels/Unsplash/Envato, photobashing 3D-model renders, compositing in Affinity Photo/Photoshop) as integral to the 'artistic process,' not preprocessing chores separate from the AI generation.

A specific creative-compromise technique is named: reshaping an idea (butterfly wings) into a form 'closer to what the AI model understands' (cat ears) while still keeping the artistic intent — i.e., deliberately meeting the model's training biases halfway rather than fighting them.

Converting the prompt textbox to an 'input' and relocating it below the image/output is presented as a deliberate ergonomic workflow choice to avoid scrolling and preserve creative flow, not just a technical convenience.

The claim that prompt wording still influences output even when the visible style doesn't change stylistically suggests prompt text and IP-Adapter-driven style operate on at least partially separable channels within the same generation.

Chaining two IP Adapters in series (rather than doing two separate generation passes) is presented as the specific mechanism for blending multiple style references within one render.

Feeding a model's own prior outputs back in as new inputs with adjusted IP Adapter weight and denoise is used as a deliberate, repeatable technique for combinatorial exploration ('go down the rabbit hole') rather than a one-off trick.

Envato Elements' 3D model feature is noted as useful for photobashing specifically because of its background-removed, angle-specific renders, despite the presenter stating they are 'personally not a fan' of the platform and noting you can't adjust lighting, camera angle, or distance.

«this is inside of comu ey I tried to make it happen in automatic 1111 it didn't work in the same way and also in comu ey I have full control over the process and full control over where the elements are so that I can stay in the creative flow»

— 01:40

«art is difficult art is complex art is the search for a solution for a complex problem that is your own expression so pressing one button is not the way to do that»

— 02:08

«working through the process finding your own Solutions building your own machines and train your own models that is Artistic process»

— 02:21

«the task here that we want to achieve is to turn our input our vision our idea into a more creative output by applying our concept and the style that we like and put them together into a final work»

— 03:28

«set the value pretty high at 1.25 and the start at zero and the end at one so it is there for the full process»

— 10:16

«I want to highly encourage you to go down the rabbit hole»

— 14:23

Reception

Strong audience support for ComfyUI and AI-assisted art, with engaged community sharing positive experiences and philosophical arguments defending artistic merit despite minimal pushback.

As a technique-dense workflow tutorial, the video is thorough and specific about node settings (models, weights, sampler values) and grounds its complexity in an explicit stated philosophy that manual compositing plus multi-stage IP-Adapter/ControlNet chaining constitutes the 'artistic' work, though its walkthrough leans on the viewer already knowing IP Adapter/ControlNet installation, which it defers to a separate linked video.

15:13

↳ Olivio Sarikas · YouTube

Watch original