Deploying Stable Diffusion based on LoRA fine-tuning [free trial on Alibaba Cloud]

Introduction to Stable Diffusion

Stable Diffusion is a text-to-image latent diffusion model built in collaboration between Runway and the University of Munich, the first version of which will be released in 2021. The current mainstream versions include v1.5, v2 and v2.1. It is mainly used to generate detailed images based on text descriptions, but is also applied to other tasks such as inpainting images, generating image-to-image translation guided by text cues, etc. This article mainly introduces how to deploy the Web-UI of Stable Diffusion in Alibaba Cloud Interactive Modeling (PAI-DSW)
for free , and fine-tune Stable Diffusion based on LoRA.

The deployment effect is shown in the figure below:

Environment and resource preparation process

Trial of Interactive Modeling (PAI-DSW)

Before using, you need to register on Alibaba Cloud and log in to your account: https://free.aliyun.com/ After
logging in, select the machine learning platform API in the product category on the left

or click the link directly , you can see Go to the 3 trial products provided by Alibaba Cloud in the machine learning platform PAI.

Select Interactive Modeling-PAI-DSW and click Try Now.

Choose to agree to the service agreement, and click Try Now.

Click Activate PAI and create a default workspace.

Click Complete Authorization (I have authorized it here), and click Confirm to open a contract and create a default workspace. Note: There is no option to activate OSS here. If you need long-term storage, you need to activate OSS (there may be a fee).

It will display that the activation is complete, click to enter the PAI console.

Click on the created workspace in the PAI console.

After entering, click Interactive Modeling on the left , and click Create Instance to start creating an instance.

Create an instance in the created workspace

Fill in the instance name and select the GPU specification. The specification name is ecs.gn7i-c8g1.2xlarge.

Select stable-diffusion-webui-env:pytorch1.13-gpu-py310-cu117-ubuntu22.04 in the official image, click Next, and then click Create Instance.
Note : It takes some time to create a DSW instance, usually about 2 to 15 minutes.

After the creation is successful, click Open to enter the PAI-DSW development environment.

Web-UI deployment of Stable Diffusion

In the launcher of the AI-DSW development environment, select DSW Gallery

under Tool to find or search for Stable Diffusion WebUI usage examples, and click to open in DSW .

Download the stable-diffusion-webui open source library and other dependencies

Run the corresponding commands in the notebook to download the stable-diffusion-webui open source library and other dependencies.

After running the command, as shown in the figure below, it means that the download has been completed.

Install common plugins

Run the corresponding command in Notebook, install tagcomplete and Chinese plugin.

After running the command, as shown in the figure below, it means that the installation is complete.

download model

Use the open source SD model Counterfeit-v2.5 as the base model.
Run the corresponding command in Notebook to download the model.

Start WebUI in DSW

Run the corresponding command in the Notebook to start the WebUI service.


In the returned result, click the URL link (http://127.0.0.1:7860) to enter the WebUI page, where inference can be performed.

Configure the following parameters on the Vincent diagram page:

• 正向prompt：
  ((masterpiece,best quality)),1girl, solo, animal ears, rabbit, barefoot, knees up, dress, sitting, rabbit ears, short sleeves, looking at viewer, grass, short hair, smile, white hair, puffy sleeves, outdoors, puffy short sleeves, bangs, on ground, full body, animal, white dress, sunlight, brown eyes, dappled sunlight, day, depth of field
• Negative prompt:
• EasyNegative, extra fingers,fewer fingers
• Sampling method: DPM++2M Karras
• HD Repair: check
• Redraw Amplitude: 0.6
• Magnification: 1.8
• Height: 832
• Prompt Word Correlation (CFG Scale): 10
  Other related parameters can also be set as required.

Click Generate to output the inference results as shown in the figure, and support the saving of pictures.

Fine-tuning of Stable Diffusion and Web-UI deployment

Install Diffusers

Create a new cell in Notebook and download the Differs warehouse to be used

! git clone https://github.com/huggingface/diffusers
! cd diffusers && git checkout e126a82cc5d9afbeb9b476455de24dd3e7dd358a
! cd diffusers && pip install .

Verify that the installation was successful

import diffusers

Configure accelerate.

! mkdir -p /root/.cache/huggingface/accelerate/
! wget -c http://pai-vision-data-sh.oss-cn-shanghai.aliyuncs.com/aigc-data/accelerate/default_config.yaml -O /root/.cache/huggingface/accelerate/default_config.yaml

The accelerate configuration is successful.

Install related dependencies

! cd diffusers/examples/text_to_image && pip install -r requirements.txt 

Fine-tuning the Stable Diffusion model

Execute the following code in the Notebook to download the clothing dataset and the code based on LoRA fine-tuning.

! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/cloth_train_example.tar.gz && tar -xvf cloth_train_example.tar.gz
! wget http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/try_on/train_text_to_image_lora.py

After running the command, as shown in the figure below, it means that the download has been completed.

Execute the following code in Notebook to see the sample picture

from PIL import Image
display(Image.open("cloth_train_example/train/20230407174421.jpg"))

Execute the following code in Notebook, download the pre-trained model and convert it into diffusers format.

safety_checker_url = f"{prefix}/aigc-data/hug_model/models--CompVis--stable-diffusion-safety-checker.tar.gz"
aria2(safety_checker_url, safety_checker_url.split("/")[-1], "./")
! tar -xf models--CompVis--stable-diffusion-safety-checker.tar.gz -C /root/.cache/huggingface/hub/

clip_url = f"{prefix}/aigc-data/hug_model/models--openai--clip-vit-large-patch14.tar.gz"
aria2(clip_url, clip_url.split("/")[-1], "./")
! tar -xf models--openai--clip-vit-large-patch14.tar.gz -C /root/.cache/huggingface/hub/

model_url = f"{prefix}/aigc-data/sd_models/chilloutmix_NiPrunedFp32Fix.safetensors"
aria2(model_url, model_url.split("/")[-1], "stable-diffusion-webui/models/Stable-diffusion/")

! python diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py \
--checkpoint_path=stable-diffusion-webui/models/Stable-diffusion/chilloutmix_NiPrunedFp32Fix.safetensors \
--dump_path=chilloutmix-ni --from_safetensors

After running the command, as shown in the figure below, it means that the file has been downloaded and converted.

Model fine-tuning. Set num_train_epochs to 200, and run the following code for LoRA-based fine-tuning training. After the fine-tuning is completed, the model file will be generated in the cloth-model-lora directory

! export MODEL_NAME="chilloutmix-ni" && \
export DATASET_NAME="cloth_train_example" && \
accelerate launch --mixed_precision="fp16" train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME --caption_column="text" \
  --width=640 --height=768 --random_flip \
  --train_batch_size=1 \
  --num_train_epochs=200 --checkpointing_steps=5000 \
  --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 \
  --seed=42 \
  --output_dir="cloth-model-lora" \
  --validation_prompt="cloth1" --validation_epochs=100

You can directly see fine-tuning related information, such as epochs, batch_size, etc. Due to the small sample size, there are only 11 samples, and the fine-tuning process takes about 16 minutes.

During the training process, you can open Terminal and enter the nvidia-smi command to check the video memory usage.

It can be seen that the video memory occupies about 14G.

Prepare the model files required by WebUI

Convert the lora model file into a format supported by WebUI and copy it to the directory where WebUI is located.

! wget -c http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/EasyCV/datasets/convert-to-safetensors.py
! python convert-to-safetensors.py --file='cloth-model-lora/pytorch_lora_weights.bin'
! mkdir stable-diffusion-webui/models/Lora
! cp cloth-model-lora/pytorch_lora_weights_converted.safetensors stable-diffusion-webui/models/Lora/cloth_lora_weights.safetensors

Prepare additional model files. In order to speed up the download, you can run the following command to directly download additional model files.

detection_url = f"{prefix}/aigc-data/codeformer/detection_Resnet50_Final.pth"
aria2(detection_url, detection_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
parse_url = f"{prefix}/aigc-data/codeformer/parsing_parsenet.pth"
aria2(parse_url, parse_url.split("/")[-1], "stable-diffusion-webui/repositories/CodeFormer/weights/facelib/")
codeformer_url = f"{prefix}/aigc-data/codeformer/codeformer-v0.1.0.pth"
aria2(codeformer_url, codeformer_url.split("/")[-1], "stable-diffusion-webui/models/Codeformer/")

embedding_url = f"{prefix}/aigc-data/embedding/ng_deepnegative_v1_75t.pt"
aria2(embedding_url, embedding_url.split("/")[-1], "stable-diffusion-webui/embeddings/")

model_lora_url = f"{prefix}/aigc-data/lora/koreanDollLikeness_v10.safetensors"
aria2(model_lora_url, model_lora_url.split("/")[-1], "stable-diffusion-webui/models/Lora/")

Start WebUI in DSW

Execute the following command in Notebook to start WebUI.

! cd stable-diffusion-webui && python -m venv --system-site-packages --symlinks venv
! cd stable-diffusion-webui && \
  sed -i 's/can_run_as_root=0/can_run_as_root=1/g' webui.sh && \
  ./webui.sh --no-download-sd-model --xformers

In the returned result, click the URL link (http://127.0.0.1:7860) to enter the WebUI page.

After clicking the link, inference can be performed on the WebUI page.

Configure the following parameters on the Vincent diagram page:

• Prompt：cloth1,lora:koreanDollLikeness_v10:0.4, (extremely detailed CG unity 8k wallpaper),(RAW photo, best quality), (realistic, photo-realistic:1.2), a close up portrait photo, 1girl, shopping mall rooftop cafe, outdoor, smile, (high detailed skin:1.4), puffy eyes, gorgeous hair, air bangs, brown black hair, soft lighting, high quality,lora:cloth_lora_weights:1

• Negative prompt：ng_deepnegative_v1_75t,paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), (grayscale:1.2), skin spots, acnes, skin blemishes, age spot, glans,extra fingers,fewer fingers,(watermark:1.2),(letters:1.2),(nsfw:1.2),teeth

Sampling method (Sampler): Euler a
sampling iteration steps (Steps): 50
Width and height : 640, 768
Other related parameters can also be set as required.

Click Generate to output the inference results as shown in the figure, and support the saving of pictures.

Note
After receiving the free resource pack, please use it within the free quota and valid trial period. If you continue to use computing resources after the free quota is exhausted or the trial period ends, a postpaid bill will be generated. You can go to the resource instance management page
to check the resource package usage .

If you do not need to continue using the DSW instance, you can follow the steps below to stop the DSW instance.

• Log in to the PAI console .

• Click the workspace list in the left navigation bar, and click the default workspace name on the workspace list page to enter the corresponding workspace.

• In the left navigation bar of the workspace page, select Model Development and Training > Interactive Modeling (DSW) to enter the Interactive Modeling (DSW) page.

• Click Stop under the Operation column of the target instance to stop resource consumption after a successful stop.