Detailed Tutorial for ControlNet Guided Image Generation

foreword

Stable Diffusion is currently a big killer in the field of image generation. The goal of ControlNet is to add additional conditions to control the final generated image, including edge detection, depth estimation, segmentation, pose estimation, graffiti and other functions.

Stable diffusion WebUI

Configuration Environment

First, you need to install Anacomda or Miniconda software to facilitate the management of our virtual environment. Here, everyone has installed this software by default.

Open Anaconda Prompt or Miniconda, create a new virtual environment named sd_webui, where the python version must be 3.10.6, otherwise it may fail. Here you need to configure the mirror source, let me show you the contents of my C://user/***/.condarc file.

channels:
  - https://mirrors.ustc.edu.cn/anaconda/cloud/msys2/
  - https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge/
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/free/
  - https://mirrors.ustc.edu.cn/anaconda/pkgs/main/
  - defaults
show_channel_urls: true

If there is no mirror source configured, you can directly replace the original content, and then create a virtual environment.

conda create -n sd_webui python=3.10.6

It will prompt y/n in the middle, just enter y here and press Enter.

Wait for the virtual environment to be successfully created and then enter the environment.

activate sd_webui

upgrade pip

python.exe -m pip install --upgrade pip

Change the default download path of pip to the Tsinghua source mirror. There are many mirror sources here, which can be changed according to the actual situation.

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

In addition, CUDA needs to be installed, but I didn't install it here, because my computer had CUDA before running deep learning itself. I believe that students who have configured deep learning should be able to install it soon. It should be noted that my CUDA version is 11.1. Although the pytorch corresponding to CUDA11.7 was downloaded during the installation process, it can still run successfully.

Download stable diffusion webui

Install Git, everyone has already installed this software by default, official website: https://git-scm.com/download/win
use Git to clone the stable diffusion webui warehouse from the official website, this code does not need to be run in Git, directly in Anaconda Prompt or You can also use Git commands in Miniconda. At this time, the warehouse will be cloned to the current path by default. For example, in the case below, it will be cloned to the root directory of the H drive.

Clone the stable diffusion webui warehouse, of course, you have to surf the Internet scientifically when cloning.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

Just wait for the download to the local, the following errors may occur during the process:

OpenSSL SSL_read: Connection was reset, errno 10054
fatal: unable to access 'https://github.com/AUTOMATIC1111/stable-diffusion-webui.git': Failed to connect to github.com port 443: Connection refused
Failed to connect to github.com port 443 after 21100 ms: Timed out
or the cloning process is very slow

Methods that can be tried at this time include but are not limited to the following:

# 关闭证书校验
git config --global http.sslverify "false"

# 取消全局代理
git config --global --unset http.proxy
git config --global --unset https.proxy

# 换服务器的节点
# 配置Git的端口号

Explain the last method, this method can solve the problem that cloning is still very slow even if Science is online. First of all, because we use a proxy server here, open the network settings to the proxy, you can find that there are two information of address and port.

Remember the address and port, and use the code below to configure Git.

git config --global http.proxy 127.0.0.1:10809

Then you can find that the speed of Git cloning is directly full, which is much more pleasing to the eye than the previous tens of kb.

If done is displayed, the clone is successful. Of course, the above methods may not be able to solve all problems. If you encounter unsolvable problems or solutions in the actual process, please let me know.
Now open the root directory of the H disk that you just cloned, and you will find that the folder stable-diffusion-webui has appeared. The specific drive letter or path should be modified according to the actual situation.

Next, download the pre-training model of stable diffusion, official website: https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main , click Files and versions, download the file drawn by the red line, any one should be Can. Here I downloaded the files marked in blue.

After the download is complete, put it in the path of D:\stable-diffusion-webui\models\Stable-diffusion.

Install torch and torchvision

Enter the path of stable diffusion webui in Anaconda Prompt or Miniconda

D:
cd stable-diffusion-webui

Run the webui-user.bat file

At this point, the launch.py ​​file under the stable diffusion webui will be executed first, which is to install torch, torchvision, some other packages and clone some github warehouses. Because this process will definitely not succeed once, let me first explain that during the command line operation of windows, you can use CTRL+C twice to exit the process.
Open launch.py ​​to the prepare_envitonment function and you can find that the required torch and torchvision versions are

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117

Since the torch package is relatively large, if the online installation fails, you can also install it offline, that is, download the corresponding .whl file first, official website: https://download.pytorch.org/whl/torch_stable.html, find the one starting with cu117 Path, download the corresponding python3.10 torch and torchvision.

After the download is complete, put it under the path of stable diffusion webui, and install it with pip. Note that you must install torch first and then torchvision.

pip install torch-1.13.1+cu117-cp310-cp310-win_amd64.whl
pip install torchvision-0.14.1+cu117-cp310-cp10-win_amd64.whl

Online installation or offline installation can be determined according to the actual situation. After the installation is complete, run webui-user.bat again. If you still enter the steps of installing torch and itorchvision, just go to the launch.py ​​file and comment out this line.

If the above is done, after running webui-user.bat, some other packages will be installed first, and then Installing gfpgan will appear, indicating that the above installation has been successful.

Clone GFPGAN, CLIP, open_clip, stable_diffusion, taming-transformers, k-diffusion, CodeFormer, BLIP and other warehouses

Generally, there is a high probability of failure when running to install these warehouses, but if many problems of Git cloning have been solved before, it should be successful once.
Here is how to solve it when it is unsuccessful. The unsuccessful error is:

RuntimeError: Couldn't install ***

Here I solved it by setting the Git port before. If it can't be solved, there are two solutions here. It is recommended to read it first and try again.

1. We can use the method of git cloning first to clone these packages and then copy them to the D:\stable-diffusion-webui\repositoriespath. Here we must follow launch.pythe installation order in the file. The following

Xformer is an optional package, which can be installed or not. It should be a graphics card If the video memory is not large, it is better to install it. Here we only talk about the package at the back. These statements are followed by the corresponding github link. If there is a @************** after the corresponding link, delete @ and all the strings behind it, and remember to save it.

All cloned using Git, as follows

git clone https://github.com/TencentARC/GFPGAN.git
git clone https://github.com/openai/CLIP.git
git clone https://github.com/mlfoundations/open_clip.git
git clone https://github.com/Stability-AI/stablediffusion.git
git clone https://github.com/CompVis/taming-transformers.git
git clone https://github.com/crowsonkb/k-diffusion.git'
git clone https://github.com/sczhou/CodeFormer.git'
git clone https://github.com/salesforce/BLIP.git'

If the current path is not modified in Git, all files will be cloned to C:\Users\***the path by default.

Since the folder name directly cloned is inconsistent with that in the launch.py ​​file, it is necessary to change the folder name here, as shown in the figure. After modifying the name,

copy and copy several packages in the picture to D:\stable-diffusion-webui\repositoriesthe path. It should be noted that the three packages of GFPGAN, CLIP and open_clip are not copied here, because the installation methods of these three packages are different.
First copy the three packages to D:\stable-diffusion-webui\venv\Scriptsthe path:

cd to the path of the three packages in Anaconda Prompt or Miniconda, for example, the path of GFPGAN is: D:\stable-diffusion-webui\venv\Scripts\GFPGAN, enter respectively

cd D:\stable-diffusion-webui\venv\Scripts\GFPGAN
D:\\stable-diffusion-webui\venv\Scripts\python.exe setup.py build install
cd D:\stable-diffusion-webui\venv\Scripts\CLIP
D:\\stable-diffusion-webui\venv\Scripts\python.exe setup.py build install
cd D:\stable-diffusion-webui\venv\Scripts\open_clip
D:\\stable-diffusion-webui\venv\Scripts\python.exe setup.py build install

This will install all the packages.

2. The above packages cloned using Git actually do not need to use the clone method, you can also directly download the .zip and decompress it. The reason for using Git is to facilitate subsequent updates.

But it should be noted that after decompressing the directly downloaded .zip file, the name of the package will have an extra -main or -master, for example,

when entering the path, it is different from the previous name.
At this time, running webui-user.bat again will complete the installation of these large packages, but install some other dependencies. After the installation is complete, information such as the path and configuration file of the current weight loading will be output.
If a local port number appears, it means that the entire installation process has been successful, and the most troublesome part is over at this point.

Copy this local connection to the browser and open it, and you will enter the webui interface.
The stable diffusion checkpoint in the upper left corner will show the model we downloaded before. If it does not appear, you need to check whether it is in the right place. Note that it is placed D:\stable-diffusion-webui\models\Stable-diffusionunder the path.

Let's take a small test first to see if we can successfully generate pictures. Enter Sailor Moon in the txt2img box, this is the meaning of Sailor Moon, click the Generate button on the right to start generating, and see how the generated results are. If the configuration of the graphics card is poor, the whole process may take a long time.

The actual measurement found that about 6.3g of video memory is needed during the entire generation process, and about 5.1g of video memory is occupied in the space state. If you want to generate multiple pictures at one time, you can modify the value of the batch count parameter, for example, I set it to 4 here.

All the results will be saved D:\stable-diffusion-webui\outputsunder the path. For example, I use txt2img to generate pictures on April 1, 2023, then the results will be saved under D:\stable-diffusion-webui\outputs\txt2img-images\2023-04-01the path. Therefore, you can go here to copy the picture directly, and if there are too many generation times, you should pay attention to cleaning up these outputs.

Install the ControlNet plug-in

Click extensions on the webui interface, then click Available, enter controlnet in the search box below and press Enter.

Find sd-webui-controlnet manipulations, click install on the right to install the plugin.

After the installation is complete, click the Reload UI at the bottom to reload the page.

After reopening the page, you can find that there is an additional line of ControlNet under the original seed parameter. Click the small arrow on the right to use ControlNet.
But at this time, there is no way to use it directly. It can be found that the Model parameter is empty.

Because ControlNet also needs to download a special model, otherwise there is no way to guide these special painting functions. Here we first take Scribbles as an example, that is, use a white outline image with a black background to graffiti.
Enter the model download official website: https://huggingface.co/lllyasviel/ControlNet/tree/main/models , select the control_sd15_scribble.pth
file and click download. There are 8 models that can be selected, and the size of each parameter file is 5.71g.

After entering the next page, click download to start the download.
After the download is complete, first D:\stable-diffusion-webui\modelscreate a new ControlNet folder in the path. It would be better if this folder already exists. Then put the parameter file you just downloaded into this folder.

Return to the controlnet of the webui, click the blue button on the right to refresh the parameter file, and select the parameter file we want to use for graffiti in the updated list.

Here I found a silhouette of Sailor Moon from the Internet, directly drag it here or click to upload our picture.

Enable,
Several parameters need to be set before generating, Enable, inver input color must be selected,

here I write girl, amine in the txt2img box. Generally, the batch count is set a little more, because not all pictures are satisfactory. Take a look at the output here.

Generally speaking, it's okay, let's use another line draft to generate it.


In some pictures, the ear is also regarded as an eye, which is a bit scary to be true. Canvas size, image size, etc. can also be controlled during the actual generation process. Of course, there are also Sinicization packages that can be used for Sinicization, but sometimes it is not easy to understand the original meaning after Sinicization, so I will not Sinicize here. If you need it, you can go to other places to find tutorials.
Well, so far our tutorial is over.