It is difficult to accurately control images in AI painting, but with the birth of ControlNet, this problem has been greatly alleviated. Today I will share with you a method of using Stable Diffusion WebUI + OpenPose ControlNet to copy the poses of people in photos. You can refer to the picture above for the effect.
OpenPose can control the posture of the human body, facial expressions, and sometimes can solve the problem of bad hands (don't have high expectations, we will introduce it below).
OpenPose can directly copy the human posture of a picture, just like the title picture, or it can generate a human body picture through a skeleton diagram (what is a skeleton diagram? I will introduce it below).
Let's take the "Vincent Picture" as an example to see how to use OpenPose.
Install ControlNet
The general Stable Diffusion WebUI integration package or image has this plug-in installed. If yours has not been installed yet, you need to install it first. If you have already installed it, just skip this section.
The method is to open the Stable Dissusion WebUI front-end page, enter "Extended Plug-in" - "Install from URL", enter in the "git repository URL of the extension plug-in": https://github.com/Mikubill/sd-webui-controlnet, and then Click "Install" and restart SD WebUI after the installation is complete. Various ControlNet models and preprocessors are generally automatically downloaded and installed upon first use.
If access to github is not smooth, you can also download the plug-in package and corresponding model through other methods, and then deploy it to stable-diffusion-webui yourself. At the end of the article, I provide the plug-in package and various models I compiled.
The specific deployment path is as follows:
- ControlNet插件：<stable-diffusion-webui>/extensions
- ControlNet model：<stable-diffusion-webui>/extensions/sd-webui-controlnet/models
- ControlNet preprocessor: <stable-diffusion-webui>/extensions/sd-webui-controlnet/annotator/downloads
Note that you need to replace <stable-diffusion-webui> with your own SD WebUI deployment path. If your SD WebUI startup command configures the path of the model or makes a soft link to the folder, you also need to adjust it according to the actual situation.
After the installation is successful, the ControlNet area will appear at the bottom of the "Vincent Chart" and "Tu Shengtu" interfaces. Click the expand button on the right. There are 4 ControlNet units below, which means we can use 4 ControlNet models at the same time to control the drawing. The effect is generally enough.
If your ControlNet only has 1 unit, or you want to modify the number of this unit, you can go to "Settings" to adjust it, as shown in the figure below, find "Multiple Control Networks: Maximum Number of Models", and don't forget to restart after changing it.
Copy picture pose
When entering the "Wen Sheng Diagram" interface, you only need to select a large model you like and fill in a simple prompt word. At the end of the article, I provide a collection of models that I have compiled for reference if you are interested.
Then set OpenPose in ControlNet at the bottom of the page and follow the steps shown in the figure below:
- Here you only need to set up 1 ControlNet unit, just select the first one.
- Upload the image of the pose you want to copy.
- Enable this ControNet unit. Generally, it will be automatically checked after selecting a picture.
- Select "Perfectly match pixels" and let ControlNet automatically calculate a reasonable internal image processing space, involving underlying technical details. Just check it without thinking.
- Control Type Select OpenPose.
- The preprocessing will be automatically brought out without modification. It is used to extract skeleton images from reference images.
- The model will be brought out automatically without modification. It controls the human posture based on the skeleton diagram when generating pictures.
- This button can bring the size of the reference picture to the "Vincent Picture" parameters. If the height and width of the reference picture do not exceed 1024, it is recommended to use it. Pictures exceeding 1024 will be slower to produce. You can manually set the "Vincent Picture" according to the picture ratio. height and width parameters.
Finally, click "Generate" and wait for the picture to appear. Two pictures will be output here, one is a picture generated using a large model and OpenPose, and the other is a skeleton diagram generated based on the reference picture. We can copy this skeleton diagram and continue to use it in the future or share it with others.
Note that there is a bit of hand shape in this skeleton diagram, so theoretically, if we depict the shape of the hand in the skeleton diagram, the hand in the generated picture will be subject to certain constraints. Here is another photo of the president waving. This hand looks pretty good, but it was selected after generating many photos.
Drawing parameters:
a man in a suit waves from a plane window as he stands on the deck of a plane with his hand up, Beeple, sunglasses, a stock photo, american romanticism
Negative prompt: EasyNegative,bad hands,bad fingers
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 289185078, Size: 656x432, Model hash: c0d1994c73, Model: realisticVisionV20_v20, Denoising strength: 0.7, Clip skip: 2, ControlNet 0: "preprocessor: openpose_full, model: control_v11p_sd15_openpose_fp16 [73c2b67d], weight: 1.5, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, -1, -1)", Hires upscale: 2, Hires upscaler: Latent, TI hashes: "EasyNegative: c74b4e810b03", Version: v1.5.1
Use skeleton diagram poses
When we used OpenPose in the previous section, we first extracted the skeleton diagram of the character in the picture, and then used the picture generated by the skeleton diagram, so as long as there is a skeleton diagram, the picture can be generated. This is so convenient for creation since there aren’t that many pictures to refer to.
For the convenience of demonstration, the skeleton diagram extracted in the previous section is used directly. We just need to change the settings here in ControlNet, as shown below:
- Upload photos using skeleton images.
- The preprocessor is set to None because we no longer need to extract the skeleton of the person from the photo.
All other parameters remain unchanged, just generate it directly.
Create a skeleton diagram
Where did the skeleton diagram come from? Here we introduce a website and two plug-ins, both of which can generate or obtain skeleton diagrams.
Avatar Pose Maker
This is a website for making skeleton diagrams. You can adjust the posture of the human body, the perspective of taking pictures, and you can also download skeleton diagrams made by others.
Visit address: Avatar Pose Maker
The page that opens is as shown below. Let me briefly talk about the following operation areas:
- Select here the body part you want to adjust
- Here you set the parameters of specific body parts and adjust the posture.
- Here you can adjust the character's perspective, whether it is front, side, looking up, looking down, etc.
- Download the photo of the skeleton diagram you created here
- Many human poses can be randomly generated here.
- Here you can enter and view the postures shared by others. There are labels explaining what the postures mean. Just choose to download.
OpenPose Editor
This is a plug-in for Stable Diffusion WebUI. The plug-in access address is: https://github.com/fkunn1326/openpose-editor.git . For the plug-in installation method, please refer to "Installing ControlNet" above.
After successful installation, we can see "OpenPose Editor" in the first-level tab. The use of this plug-in is relatively simple. You can directly adjust the posture of the character in the drawing area, and then download the picture for later use, or directly send it to Wensheng Pictures, Pictures Use for drawings, etc.
Here is the rendering of my skeleton diagram:
Drawing parameters:
eastern girl,18 years old, standing, waving left hand, balck hair, long straight hair, blue eyes, white shirt, blue jeans, trees, blue sky, cloud
Negative prompt: EasyNegative, bad hands, bad fingers, malformed
Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1995058330, Face restoration: CodeFormer, Size: 512x512, Model hash: 15012c538f, Model: realisticVisionV51_v51VAE, Denoising strength: 0.7, Clip skip: 2, ControlNet 0: "preprocessor: none, model: control_v11p_sd15_openpose_fp16 [73c2b67d], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: False, control mode: Balanced, preprocessor params: (-1, -1, -1)", Hires upscale: 2, Hires upscaler: Latent, TI hashes: "EasyNegative: c74b4e810b03", Version: v1.5.1
3D Openpose
This is another Stable Diffusion WebUI plug-in. Access address: https://github.com/nonnonstop/sd-webui-3d-open-pose-editor.git . For students who don’t know how to install it, please refer to "Installing ControlNet" above. ".
After successful installation, we can see "3D Openpose" in the first-level tab. This plug-in is relatively powerful. We can adjust the shape of characters in a 3D environment, but the operation will be more complicated. You can experience it yourself. Down.
- Double-click a joint in the skeleton diagram to adjust the direction of the joint.
- The perspective and size of the character can be adjusted by sliding the mouse.
- In particular, it supports depth maps and edge detection maps of hands and feet, which are used to optimize the generation of hands and feet. You can also generate only hands separately.
After making adjustments, click "Generate".
The generated preview pictures are as follows. The first picture is the skeleton picture, and the following pictures are auxiliary pictures used to draw hands and feet.
You can send it directly to Wenshengtu and Tushengtu, or you can download it for later use. It doesn't feel very useful to send it directly. It is recommended to download it first and then upload it. The setting method of ControlNet is the same as the above method of directly using the skeleton diagram to generate images. Note that there is no need to select a preprocessor.
If we want to make special repairs to our hands and feet, we can use the auxiliary drawings for drawing hands and feet. Select the ControlNet for edge detection here. Be careful not to select the preprocessor. You can adjust the number of start control steps and the number of end control steps according to the actual situation. Do not go from 0 to 1 in the Vincent diagram, as it will affect the structure of the picture. In the diagram, Hand and pedicure can be done using partial drawing.
Although we specifically dealt with the hands and feet, deformed hands and feet often appear in the output. The effect is there, but the gap with expectations is still relatively large. Here are two examples:
- Vincent picture tape repairer:
- First, the text produces the picture, and then the picture produces the partial hand repair:
Download
If it is inconvenient for you to download plug-ins and models, you can use the resource package I compiled, follow the public account: Yinghuo Walk AI (yinghuo6ai), and send a message: plug-in to get the download address.
The above is the main content of this sharing. If you have any questions, please feel free to communicate.
If you haven’t used Stable Diffusion WebUI yet, you can read these articles first to learn how to use it:
Teach you step by step how to make alchemy in cloud environment (deploy Stable Diffusion WebUI)