Article directory
Introduction and installation of ControlNet
Introduction to ControlNet
ControlNet in Chinese means control network. It is essentially an extension plug-in for Stable Diffusion. It was developed by researchers at Stanford University in February 2023 and allows creators to easily control objects in AI images and videos. It will control image generation based on various conditions such as edge detection, sketch processing, or human pose. ControlNet can be summarized as a simple stable diffusion fine-tuning method.
Why do you need ControlNet?
Because before the emergence of ControlNet, "AI painting" based on the diffusion model was very difficult to control. The process of drawing or drawing was full of randomness. If you just simply entertain yourself, this randomness is not It will cause a lot of trouble, and deviations in the drawn content are acceptable, but for some scenes with specific needs, if precise "control" cannot be achieved, you can only rely on randomness to get what you want. As for pictures, stability and effectiveness cannot be guaranteed.
ControlNet implements many control methods in a dimensionally reduced way and is quite stable. ControlNet and Lora have many similarities. They are both additional networks that fine-tune the diffusion model, that is, they provide clear guidance for the generation of the diffusion model based on some additional input information.
The difference between ControlNet and Prompt: If you just enter dancing (dancing) in prompt, the generated pictures will have a variety of dance postures (even with more words as guides). The essence of ControlNet is that you can input it a picture that records a specific "posture" information to guide the drawing.
ControlNet installation
Find Extensions in the Stable Diffusion WebUI interface to install, enter the following link in the corresponding text box:
https://github.com/Mikubill/sd-webui-controlnet
If an installation error is reported, you can manually download and install it
. After the installation is completed, You need to click Check for updates and Apply and restart UI
. If the UI is not loaded, you can restart the Stable Diffusion WebUI.
After successful installation, it can be opened as shown in the following figure:
ControlNet function introduction
After the installation is complete, let's take a look at each part of ControlNet and how each configuration affects the output.
Enable : Check this box to enable ControlNet. (Enabling ControlNet requires an additional image processing process, so it will occupy a certain amount of video memory.)
Low VRAM: This will slow down the ETA process, but helps to use less computing space (VRAM is recommended for video memory less than 6 GB)
Pixel Perfect (Perfect Pixel): It can automatically calculate the most suitable resolution for the image produced by the preprocessor to avoid image blur and deformation caused by inappropriate size. (It is recommended to select)
Allow Preview will open a small preprocessing window, which is specially prepared for preprocessing images.
Control type.
Preprocessor: Preprocessor
Model: The model
mainly controls the basic process of ControlNet. The most important thing to pay attention to is Control Weight, which is the control weight. It is the same as the weight of the prompt word/Lora. It will determine how the control effect is presented in the picture. The intensity that comes out usually uses the default value of 1. The following Starting Control and Ending Control Stop will determine at what stage in the process of generating the image the control effect is added. The default 0 to 1 means that it will take effect from beginning to end. You can try to set it. different values.
Control mode: You can control which one is more preferred between the influence of the prompt word and the information of ControlNet. Balanced is usually used by default.
ControlNet applications and demonstrations
Model download: https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main
Download to extensions/sd-webui-controlnet/models. When downloading, be sure to download the configuration file (.yaml) corresponding to the model. )
What needs to be noted here is that if there is a duplicate file yaml under models, just choose to overwrite it.
1. Canny edge detection
Canny detects input images by using edge detectors to create outlines of high-contrast areas. Lines can capture very detailed information, but if there are some objects in the background of your image, it will most likely detect unwanted objects. So the fewer objects in the background, the better. The best model for this preprocessor is control_sd15_canny.
2. Depth & Depth Leres
background replacement method: In the img2img image drawing mode, remove the background through the remove background function in the Depth_leres model of ControlNet, and then replace the desired background through the prompt word.
Application model: Depth, preprocessor Depth_leres.
Key points: If you want to replace the background more perfectly, you can add a mask to the image content that needs to be retained in the Inpaint mode of the image. The remove background value can be set at 70-80%.
Depth_leres example: (replace the original image background with the office background)
3、MLSD ( Mobile Line Segment Detection)
MLSD Preprocessor is best suited for generating strong lines capable of detecting architecture and other man-made works that require unique and rigid silhouettes. But it is not suitable for handling non-rigid or curved objects. MLSD is suitable for generating interior layouts or building structures as it can highlight straight lines and edges. The best model for this preprocessor is control_sd15_mlsd.
4. Three views of the character
Method: Use ControlNet's Openpose model to accurately identify the human pose, and then use prompt words and style models to generate pictures of the same pose.
Application model: OpenPose. In ControlNet version 1.1, a variety of posture detection methods are provided, including: openpose body, openpose_face body + face, openpose_faceonly face only, openpose_full body + hand + face, openpose_hand hand, which can be flexibly applied according to actual needs.
OpenPose character three-view example:
Key points: Upload openpose three views, load charturner style model (https://civitai.com/?query=charturner), add prompt words to keep the background clean (simple background, white background:1.3), multiple views
