ControlNet: Precise Image Control

01What Is ControlNet?

ControlNet is an extension for diffusion models that introduces additional control signals into the generation process. Instead of using only text as input, you can use reference images with ControlNet to control the pose, depth, edges, or composition of the generated image.

02ControlNet Types

The most important ControlNet variants:

• Canny Edge: Detects edges in the reference image. Good for architectural structures and clear outlines.
• Depth: Uses a depth map. Ideal for 3D compositions and perspective.
• OpenPose: Detects human poses (skeleton). Perfect for consistent body postures.
• Scribble: Converts simple sketches into detailed images. Low entry barrier.
• Normal Map: Uses surface normals for detailed 3D structures.
• Soft Edge (HED): Softer edge detection than Canny, for more organic results.
• Tile: For upscaling and detail improvement of existing images.
• IP-Adapter: Uses a reference image for style and content (not a classic ControlNet, but a related concept).

03ControlNet in ComfyUI

In ComfyUI, use the 'Apply ControlNet' node. Connect it to a preprocessing node (e.g., Canny Edge Detection) and the KSampler. The strength of control is managed via the 'strength' parameter (0.0–1.0). Lower values give the model more creative freedom, higher values follow the reference image more strictly.

04Combined Usage

05Practical Tips

For the best results with ControlNet:

• Start with a strength of 0.5–0.7 and then adjust
• Use the appropriate ControlNet type for your goal – Canny for structure, Pose for people
• High-quality reference images lead to better results
• Experiment with start/end parameters to apply ControlNet only in certain sampling steps
• For SDXL and Flux, there are special ControlNet models – don't use the SD 1.5 variants