Article Directory
Primer
Look at the two pictures
first. The above words are two valid QR codes. As long as you scan these images with your mobile phone, you can scan out the content. Is it unbelievable?
Explosive Art QR Code
Recently, a new form of QR codes has been spreading wildly on the Internet. They get rid of the appearance of traditional black and white squares, but present pleasing images, which are impressive at first glance.
How this artistic QR code was made
The core solution is ControlNet based on Stable Diffusion. Use ControlNet and QR Code. Convert a stylized image into a scannable QR code by positioning points.
Introduction to ControlNet
ControlNet is a stable diffusion model that can replicate composition or human pose from reference images.
Experienced users of stable diffusion know how difficult it is to generate the exact composition you want. The images are somewhat random. All you can do is play the numbers game: generate lots of images and pick one you like.
With ControlNet, Stable Diffusion users finally have precise control over the placement and appearance of their subjects!
ControlNet is a neural network model for controlling stable diffusion models. You can use ControlNet with any stable diffusion model. The most basic form of using a stable diffusion model is text-to-image. It uses text cues as conditions to guide image generation so that images matching the text cues are generated.
Limitations of ControlNet
ControlNet adds a condition in addition to the text prompt. In ControlNet, additional conditions can take many forms.
Example of edge detection
ControlNet takes an additional input image and detects its contours using a Canny edge detector. The image containing the detected edges is then saved as a control map. It is fed into the ControlNet model as an additional condition to the text prompt.
Stable Diffusion Control Network Model Workflow.
Stable Diffusion Control Nets with Canny Edge Conditions.
The process of extracting specific information (edges in this case) from an input image is called annotation or preprocessing (in ControlNet extension).
Human Pose Detection Example
Edge detection is not the only way to preprocess images. Openpose is a fast human keypoint detection model that can extract human poses such as hands, legs, and heads. See the example below.
Input image annotated for human pose detection using Openpose.
Keypoints are extracted from the input image using OpenPose and saved as a control map containing keypoint locations. This is then given to Stable Diffusion as an additional condition along with a text prompt. Images are generated based on these two conditions.
What is the difference between using Canny edge detection and Openpose?
The Canny edge detector extracts the edges of objects and backgrounds. It tends to translate scenes more faithfully. You can see the dancing men turned into women, but the outlines and hairstyles are preserved.
OpenPose only detects human key points, such as the position of the head, arms, etc. Image generation is more free, but follows the original pose.
Install the stable diffusion control network ControlNet
Let's find out how to install ControlNet in AUTOMATIC111, the popular, full-featured (and free!) stable diffusion GUI. We will use this extension, which is the de facto standard for working with ControlNet.
If you already have ControlNet installed, you can skip to the next section to learn how to use it.
Install ControlNet
Install ControlNet on a Windows PC or Mac
You can use ControlNet with AUTOMATIC111 on a Windows PC or Mac. If you don't have AUTOMATIC111 installed, follow the instructions in these articles to install it.
If you already have AUTOMATIC111 installed, please make sure your copy is up to date.
Install ControlNet Extensions (Windows/Mac)
1. Navigate to the Extensions page.
2. Select the "Install from URL" tab.
3. Put the following URL into the URL field of the extension repository.
https://github.com/Mikubill/sd-webui-controlnet
4. Click the "Install" button.
5. Wait for the confirmation message that the extension is installed.
6. Restart AUTOMATIC111.
7. Visit the ControlNet Models page.
8. Download all model files (filenames end in .pth).
(If you don't want to download them all, you can temporarily download the most commonly used openpose and canny models.)
9. Put the model files in the models directory of the ControlNet extension.
Stable diffusion webui\extensions\sd webui control net\models
Restart AUTOMATIC111webui.
If the extension installed successfully, you will see a new collapsible section called ControlNet in the txt2img tab. It should be just above the script drop-down menu.
Update ControlNet Extensions
ControlNet is a rapidly evolving extension. It is not uncommon to find outdated copies of ControlNet.
The update is only required when running AUTOMATIC1111 natively on Windows or Mac. The website's Colab notebooks always run the latest ControlNet extensions.
To determine if your version of ControlNet is up to date, compare the version number in the ControlNet section of the txt2img page to the latest version number.
Step 1: Open the Terminal application (Mac) or PowerShell application (Windows).
Step 2: Navigate to the ControlNet extension's folder. (if you install elsewhere, please adjust accordingly)
cd stable-diffusion-webui/extensions/sd-webui-controlnet
Step 3: Update the extension by running the following command.
git pull
How to use Stable Diffusion to generate artistic QR codes
A recent Reddit thread showcased a series of artistic QR codes created by Stable Diffusion. These QR codes were generated using a custom trained ControlNet model. Like another day in the stablediffusion community, people quickly learned how to generate QR codes using stablediffusion without custom models.
Generate QR code
You first need a QR code. To increase your chances of success, use a QR code that meets the criteria below.
Use a high tolerance setting (30%).
Leave white space around the QR code (quiet zone).
Use the most basic squares to fill the black and white pattern.
Avoid generators that introduce thin white lines between black elements.
We will be using this QR code generator in this tutorial.
Step 1: Select the text type and enter the text of the QR code.
Step 2: Set the fault tolerance rate to 30%.
Step 3: Press Generate.
Step 4: Download the QR code as a PNG file.
Effect
The following are some renderings that I tried to generate.
The effect is much more beautiful than the monotonous normal QR code.