AI Painting: StableDiffusion Alchemy Lora Raiders - Actual Combat Cute Pet Picture Generation

words written in front

Recently, I found many extremely cute and beautiful pictures of cute pets in Xiaohongshu, and I am deeply fascinated by these wonderful pictures

So I thought about using AI to paint StableDiffusion (hereinafter referred to as SD).

The following is the whole process of detailed practical operation, including all the materials used have been packaged to the network disk.

The final result of the last attempt is as follows:

For more pictures, please check the network disk:

"Cute pet pictures and keywords"

Link: https://pan.quark.cn/s/ba9c3e8ef92a

If you want to know more about StableDiffusion, please refer to:

AI Painting: Stable Diffusion Ultimate Alchemy Book: From Beginner to Master

The original link is more comfortable to read: AI Painting: StableDiffusion Alchemy Lora Raiders - Actual Combat Cute Pet Picture Generation

One: Prepare

The required information is packaged in the network disk, and you can download it yourself if you need it.

I shared the "StableDiffusion alchemy information" using the quark network disk

Network disk link: https://pan.quark.cn/s/3c8cc96f3221

2. The role of Lora

LORA allows us to easily draw specific characters, objects, special strokes and special style or style, which belongs to a subset of special training.

1. AI model

Create a model of your own, let this model wear your own products

2. Refining Clothes Lora

Add a Lora of clothes, you can make the character wear specific clothes

3. Change the painting style/screen background

Change the style of the photo by adding Lora, this style can be trained by yourself

What exactly is the Lora model?

professional explanation

The full name of LoRA is LoRA: Low-Rank Adaptation of Large Language Models , which can be understood as a plug-in of the stable diffusion (SD) model. Like hyper-network and controlNet, it uses a small amount of data without modifying the SD model. The data trains a painting style/IP/character to achieve customized needs, and the required training resources are much smaller than training SD models, which is very suitable for community users and individual developers. LoRA was originally applied in the NLP field for fine-tuning models such as GPT-3 (that is, the predecessor of ChatGPT). Since the number of GPT parameters exceeds 100 billion and the training cost is too high, LoRA adopts a method to only train low rank matrices (low rank matrices), and inject the parameters of the LoRA model into the SD model when using it, thereby changing the SD model. Generate styles, or add new characters/IPs to SD models.

popular explanation

Lora can reproduce the characteristics of characters and objects, fix character movements, and change the style of photos

And Lora can be trained with only a small amount of data, which is much simpler than training a large model

So you can make customized pictures by training your own Lora

But I have to say that the current Lora cannot be 100% the same, especially in terms of details.

But I believe that with the subsequent technological development, the era of Lora is not far away.

3. How to refine your own Lora model?

There are many ways to refine Lora

There are scripts for training, and some training on the website interface. Recently, many friends have made an integration package for training Lora.

There are many ways to create a Lora model, including training through scripts, operating through a web interface, and even some professionals have recently provided a one-click integration package for Lora training. Although these methods appear different on the surface, their training logic is actually the same.

Therefore, we can choose to use the integration package. The advantage of the integration package is that it integrates all the tools and steps required for alchemy into one piece of software, providing us with a more convenient and effective training method.

The integration package is to integrate all the tools needed for alchemy into one software

Alchemy is divided into the following steps:

1. Download the integration package

2. Preparations

3. Start training

4. Test Lora

5. Optimize Lora

4. Preparations before alchemy ( download integration package )

Before refining Lora, you need to confirm your computer configuration:

1. Computer configuration requires N card, and 6G memory or more

2. A card and Mac system, or a small partner with poor computer configuration is recommended to use the cloud platform

What I use here is the integration package of Juni Jiang, the up master of station B:

I shared the "StableDiffusion alchemy information" using the quark network disk

Network disk link: https://pan.quark.cn/s/3c8cc96f3221

After downloading, unzip it to the D drive or the E drive, not the C drive! !

I also tried to make Lora with Qiuye’s integration package. After comparing the two, I think Junijiang’s integration package is more suitable for Xiaobai

Open the folder after decompression, and find the "Cyberdan Furnace" application in the "cfurnace_ui" folder

You can create a desktop shortcut so you don't have to open the file every time

Once you see this page, you can install it. Click "Start Alchemy Furnace" to start refining Lora!

5. Select the appropriate large model

Just like drawing a picture, before refining Lora, you must first choose a large model to determine Lora's painting style

The cute pet trained here, I chose the "cheeseDaddys_35" large model

Real-life model = "Choose the large model of "chilloutmix"

Two-dimensional element = "select the large model of "anything"

If there is no model in your Stable Diffusion, you have to download the model first!

These two large models have been placed in the network disk for everyone.

I shared "2. Large model checkpoint" with the quark network disk

Link: https://pan.quark.cn/s/9767ac274f83

Further down, you can choose a type of Lora we want to train

After selection, it will help us choose the default parameters

Choose "Character" to train real Lora

Choose "two-dimensional" for training two-dimensional

If you want to refine the painting style, you can choose the painting style

In addition, you can also customize to refine graphic design drawings or buildings, etc.

I choose the product here

6. Production of high-quality materials

After the above preparations and parameters are set, you can start uploading your own materials and start training.

These materials are the materials we want to feed AI to learn

The quality of the material is very important! ! It will directly affect the quality of the final Lora

Our material needs to meet several points:

1. Upload 20~30 photos

2. The materials must be high-definition! ! !

3. Multi-angle photos

Here I will take the cute pet Lora as an example and upload photos of cute pets

Click "Delete All" to delete the default material

Then click "Upload Material" to upload your own photos

Under normal circumstances, the following parameters are fine by default.

If you need to adjust the parameters, it is best to understand the meaning of the parameters. If you adjust them indiscriminately, the training may fail.

Do not choose too high a resolution, it is easy to burst the video memory

In addition, if training a real Lora, you can check the "Face Strengthening Training" on the far right

After checking it, a group of photos with only faces will be cut out, so that AI can learn more facial details

7. The training process of waiting patiently

Seeing this page means that the model is being trained

At this point, just wait patiently , there is nothing to do

We can see what the following parameters mean

"Steps" is the number of training steps

A picture will appear in the lower right corner every 50 steps of training

That way you can see what Lora looks like in real time

This white hair and red skirt are keywords added in the background

Can test the generalization of Lora

Generalization is to see if this Lora can freely change the things in the photo, such as hairstyle, hair color, clothing and so on

Loss can be used to refer to the quality of the model

A good model Loss value is between 0.07 and 0.09

Note: Whether it is good or not depends on the actual test in Stable Diffusion

After the training is complete, click "Model"

You can see the generated model

According to the default parameter training, 20 models will come out, but it does not mean that the last model is the best

It is possible to refine to the sixteenth or seventeenth model is enough, and the model after that is already overtrained

So these models have to be tested in SD to know which is the best

8. How to test the quality of Lora

After the model is generated, you can go to SD to generate pictures

Such a large picture can be generated in SD, and you can directly see the effect of all models under different weights

Compare which model is better, just keep that model

Next, let's see how to generate this big picture

First copy the newly generated 10 models to the models folder of SD, and put them in the Lora folder

Note: It is best to create a new folder, such as cuteDog

Then rename the Lora without serial number code

Lora without serial number encoding is the last generated model, here is just for the convenience of testing, unify the names of all models

After saving the model, you can open SD for testing

First choose a large model

Choose which large model you use to train lora

Then enter positive keywords and negative keywords

For positive keywords, you can enter some quality words, such as the highest quality, high-definition picture quality, masterpiece, etc.

Negative keywords can directly copy what we have used before

The next step is to choose the Lora we just made

You can choose any one

Here comes the important point: replace 000017 with NUM and 1 with STRENGTH

<lora:cute dog_20230708214731-000017:1>

For example:

<lora:cute dog_20230708214731-NUM:STRENGTH>

The number of iteration steps and sampling method can be modified according to your own habits.

Then scroll to the bottom and find "Scripts"

Select " X/Y/Z Chart " in the script

Select " prompt word search/replace " for X-axis and Y-axis type

X-axis value input: NUM, 000001, 000002, 000003, 000004, 000005, 000006, 000007, 000008, 000009, 000010

The serial number here corresponds to the number of our 10 Lora

Y axis value input: STRENGTH,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1

The serial number here represents the weight of Lora

After all the parameters are set, you can click "Generate"

The generated image is such a big picture, you can see the performance of these Lora models under different intensities

9. How to optimize Lora

In fact, alchemy is a relatively metaphysical thing

Some people may be able to refine their own satisfied Lora in one go

But some people may have to practice several times to get a good Lora

At this time, we can adjust the training parameters and refine a new one.

The premise is to ensure that the materials we feed to AI are of high quality!

Otherwise, no matter how you modify the parameters, the resulting Lora will be unqualified.

Here we divide the problems encountered in refining Lora into two situations

1. The photos of Lora don’t look like me: AI didn’t learn well

2. Lora is over-fitting, and even the photos that came out collapsed: AI has learned too much

Overfitting means that no matter what keywords you input, the photos that come out are all the photos you feed to AI

There is no way to freely control the character's clothing, hairstyle, hair color, etc.

Click "Parameter Tuning" and we can set the parameters by ourselves

If the generated photo does not look like your own, you can increase the value of Repeat (number of learning steps) appropriately

If the photo is overfitting, then reduce the Repeat value

Other parameters can not be adjusted, because the default parameters are almost the optimal value

10. Resource Download Summary

Summary of Lora alchemy network disk resources: https://pan.quark.cn/s/3c8cc96f3221

StableDiffusion resource integration installation package: https://pan.quark.cn/s/2750beda9269

StableDiffusion keyword classification query: StableDiffusion keyword classification query

Summary of ControlNet data: https://pan.quark.cn/s/47bc8c79892a

Summary of AI data network disk (updated from time to time): Summary of network disk resources in AI zone

Summary of AIGC tutorials from entry to mastery: Summary of AIGC tutorials from entry to mastery

The original link is more comfortable to read: AI Painting: StableDiffusion Alchemy Lora Raiders - Actual Combat Cute Pet Picture Generation