AI - stable-diffusion artistic QR code

Series of articles:

• "Building and Using AI - stable-diffusion (AI painting)"
• "AI - Precise Control of AI Painting (ControlNet)"

1. Introduction

Recently, the use of AI painting (stable-diffusion) to artistically transform QR codes is quite popular. This idea was realized by Chinese people using the combination of Checkpoint + LoRA + QR Code ControlNet. Here are some examples of pictures:

This is the original author's article: "AI Generates Scannable Image - New ControlNet Model Demonstration"

However, the article does not describe the specific process of artistic QR codes, and there is no ready-made QR code QR Code ControlNetavailable for use (maybe it has not been published for some reason, guessing ~), some information is collected from the Internet, and the existing The ControlNet model can also achieve the same effect, let's take a look at the specific implementation steps.

Two, actual combat

Two ControlNet models need to be downloaded first:

• SD15_brightness：https://huggingface.co/ioclab/ioc-controlnet/tree/main/models
• SD15_tile：https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main

Store it in stable-diffusion-webui\extensions\sd-webui-controlnet\modelsthe directory .

Note: It is recommended to upgrade the ControlNet plug-in to the latest version, which supports more preprocessors and has a more convenient and easy-to-use UI.

1. Generate a QR code

We first prepare a QR code image, which can be generated through any of the following URLs:

• Forage QR code: https://cli.im/
• QR.io：https://qr.io/

2. Grayscale processing

1. Upload QR code
2. ControlNet Unit 0: check to enable (Enable)
3. Preprocessor:inpaint_global_harmonious
4. Model (Model):control_v1p_sd15_brightness
5. Weight (Control Weight):0.4
6. Starting and Stopping Steps (Starting Control Step, Ending Control Step): 0and1

NOTE: If you can't find inpaint_global_harmoniousthe preprocessor , update the ControlNet plug-in to the latest version.

3. Details processing

1. Upload QR code
2. ControlNet Unit 1: Check Enable (Enable)
3. Preprocessor:inpaint_global_harmonious
4. Model (Model):control_v11f1e_sd15_tile
5. Weight (Control Weight):0.6
6. Starting and Stopping Steps (Starting Control Step, Ending Control Step): 0.35and0.75

Note: If you don’t have multiple ControlNet units on the page, you can go to Settings and find Multiple ControlNets for setting. For details, see: "AI - Precise ControlNet for AI Painting (ControlNet)"

4. Configuration description

Configure according to what you want to appear on the QR code screen. For example, I want a Gundam in war to appear on the screen. My parameter configuration is as follows:

1. Positive prompt (Prompt):masterpiece, best quality, mecha, no humans, black armor, blue eyes, science fiction, fire, laser canon beam, war, conflict, destroyed city background
2. Negative prompt:UnrealisticDream, FastNegativeEmbedding
3. Sampling method:Euler a
4. Sampling steps:26
5. CFG Scale:6.5
6. Random seed (Seed):1234843297

Note: If you don't know what these configurations mean, you can refer to this article: "Building and Using AI - stable-diffusion (AI painting)"

After configuring the description, click Generatethe button and wait for the image to be generated:

So far, the artistic QR code has been generated, and it can be recognized by normal scanning~

3. Information

In the above actual combat, a new ControlNet preprocessor and two new ControlNet models appeared. What are their functions?

1. Preprocessor inpaint_global_harmonious

inpaint_global_harmoniousIt is mainly used for partial (mask part) redrawing, adding ControlNet control to the native image-based redrawing mode, so that inpaint can predict redrawing details more accurately. But note that, unlike ordinary local redrawing, ControlNet's inpaint model not only redraws the mask part, but also minor changes (redrawing) outside the mask. The advantage of this is that it can better use The redrawing area is integrated with the overall picture, making the overall drawing more harmonious and unified. Of course, you can also adjust the ControlNet guide weight to the maximum, and the ControlNet-inpaint model will try to keep the picture outside the mask area unchanged as much as possible.

The picture above is a partial redrawing of a traditional picture. The redrawn part of the original picture often has independent scenes that are incompatible with the surrounding environment, and the overall layout cannot be unified.

It can be seen that, compared with the local redrawing of the traditional image, the inpaint model of ControlNet can complete the content of the mask part under the premise of unifying the overall effect, and at the same time, the outline and details of the part outside the mask are also better. reserve.

The above introduction comes from: https://www.jianshu.com/p/2e433f71f544

2. Model control_v1p_sd15_brightness

This model brings brightness control to stable diffusion, allowing users to colorize grayscale images or recolor the resulting image.

The above introduction is from:
https://aigc.ioclab.com/sd-showcase/brightness-controlnet.html
https://huggingface.co/ioclab/control_v1p_sd15_brightness

3. Model control_v11f1e_sd15_tile

The tile model was originally designed to enlarge the image. Specifically, it is necessary to divide the original image into small areas (called tiles or tiles), and then zoom in on each tile separately, and finally gather all The enlarged tiles form a complete large picture. The reason why the splicing mode after partial enlargement is adopted is to reduce the video memory usage. However, there are also disadvantages in enlarging the picture in this way, that is, it is necessary to require that the redrawing range of each small picture should not be too large when the picture is generated, because in essence, each small picture uses the prompt words of the whole picture. When the redrawing range is too high, Older versions of the tile model are likely to generate all the content depicted by the full-image prompt in each small image.

The Tile model after ControlNet1.1 is more powerful, it has two characteristics:

• Ignore details in the image and generate new ones.
• If local tile semantics and hints do not match, global hints are ignored and diffusion is guided according to local context.

Because the model can generate new details and ignore existing image details, we can use this model to remove bad details and add refined details. For example, to remove blur caused by image resizing.

The above introduction is from:
https://www.jianshu.com/p/2e433f71f544
https://github.com/lllyasviel/ControlNet-v1-1-nightly

4. Analysis

I disassembled the pictures generated by the two ControlNet units in actual combat, and got the following process diagram:

Combined with the introduction of the two ControlNet models in the above materials, it can be roughly understood as recoloring control_v1p_sd15_brightnesson basis of the QR code image, but the QR code information will be lost. control_v11f1e_sd15_tileOn this basis, the details of the QR code information will be supplemented . As for the preprocessor inpaint_global_harmonious, because the QR code does not need to be inpainted in this scenario, it is also possible to set the preprocessor to none, which will not affect the result.