ControlNet is a very important plug-in in SD, and we will introduce this plug-in in detail below.
ControlNet
ControlNet is a neural network structure that controls the diffusion model by adding additional conditions. Simply put, ControlNet is the constraint when Stable Diffusion generates pictures, such as restricting the poses of the characters in the generated pictures.
Canny edge detection example
Workflow: Edge detection will extract the outline of the image from the input image, input it as an additional condition besides the prompt word, participate in the construction of Stable Diffusion graphics, and control the generation of graphics.
In the previous example, the background and clothes of the overall image were kept unchanged, and the color of the hair was changed. For details, please see this article: AI drawing (7) stable diffusion drawing partial redrawing
Human Pose Detection
Openpose is a fast human keypoint detection model that can extract human poses, such as the position of hands, legs, and head. For details, please see this article: AI drawing (6) stable diffusion drawing and drawing
Workflow: Keypoints are extracted from the input image using OpenPose and saved as a control map containing keypoint locations. This is then given to Stable Diffusion as an additional condition along with the text hint that the image is generated based on these two conditions.
What is the difference between using Canny edge detection and OpenPose?
Canny edge detector extracts similar edges of subject and background. It tends to translate scenes. In the previous example of AI drawing (7) stable diffusion drawing partial redrawing, you can see that the black-haired lady becomes the red-haired lady, but the outline and hairstyle are preserved.
OpenPose only detects key points of the human body, such as the position of the head and arms. Image generation is more free, but follows the original pose.
ControlNet plug-in introduction
The interface is shown in the figure below:
Related button introduction:
Enable/Enable: whether to start ControlNet
Low memory mode/Low VRAM: open when the video memory of the graphics card is not enough
Pixel Perfect: ControlNet will use the image height and width you specify in the image to generate the preprocessed image
Allow Preview: Whether the preprocessed graphics need to be previewed
Preprocessing/Preprocessor: Assuming that we input an ordinary picture, it will be processed after preprocessing. For example, when uploading a picture, the OpenPose model will process the picture into a pose map.
To upload a picture, first select [Allow Preview], then select the explosion button next to [Preprocessing/Preproessor],
The preprocessed result will be displayed in the picture box on the right
Model/Model: After we select [Preprocessing/Preproessor], next we will select [Model/Model]. ] — Correspondence,
For example;
Pose detection/openpose -- control_v11p_sd15_openpose
Line draft detection/canny -- control_v11p_sd15_canny
Detailed ControlNet model
When choosing a model, you will find that there are many ControlNet models. In order to facilitate and quickly retrieve the functions we need, we made the following mind map according to the dimension of functions. The division of functions refers to https://github.com/lllyasviel/ControlNet
Pose the human body
See [Set Human Pose], which corresponds to 5 preprocessing modules for setting human pose. We choose the preprocessing model [openpose], the model selection [control_v11p_sd15_openpose], note that the preprocessing model and the model are one-to-one correspondence.
prompt:
best quality，masterpiece,super high resolution，realistic,a girl，wide shot,full body，Long black straight hair， big blue eyes, black jeanwhite tights, adult female, asian,tree，ocean
Negative prompt:
mutated hands and fingers, deformed, bad anatomy, disfigured, poorly drawnface,mutated, extra limb, ugly, poorly drawn hands, missing limb, floatinglimbs, disconnected limbs, malformed hands, out of focus,long neck,long body
change style/color
If you want to change the style or color of the current picture, you can use canny and hed, both of which are edge detection models, and these two models will preserve the overall composition of the picture. hed retains more details than canny, it depends on personal preference.
The preprocessing model [line draft detection/canny] is used together with the model [control_v11p _sd15_canny]
The preprocessing model [softedge_hed] is used with the model [control_v11p_sd15_softedge]
It can be seen that hed retains more details than canny. The specific use depends on the individual situation.
epilogue
So, today we introduce here. Interested friends can follow us, we will continue to update this series of tutorials.
If you think it is good and helpful to you, you can follow me, and more content will be released in the future. You can also join our planet, pay attention to the official account, reply to "planet", get the invitation code, and you can join. If you don't know what our planet is, you can read this article, which has a detailed introduction.
Empowerment of AI tools to achieve efficient monetization and leave work early 
/details/132250585
I am discussing interesting topics with my friends at "AI High Efficiency [Work, Study]", why don't you come together?
https://t.zsxq.com/11KVfN0RChttps://t.zsxq.com/11KVfN0RCThe public account @AI宅player No. 2 player is an obsessed practitioner who relies on AI to help study and work. There are 20,000+ fans on the whole network, and the AI that Xiaobai can also play is continuously updated, allowing you to leave work 2 hours earlier.
What's on the planet:
e-competition column, deep learning column, AI-assisted learning column, AI tool monetization column, etc.Suitable for the crowd:
Students in school: I hope my experience can help you solve the problems you encounter.
Developers: Share your own experience, and you can broaden your path and learn more.
Hope to make money with side jobs: We use AI The way of cashing out can help you make money with sideline business
In fact, rigid standards cannot limit us with infinite possibilities, so ah! Come on guys!