4、stable diffusion

github

安装anaconda环境

conda env create -f environment.yaml
conda activate ldm

安装依赖

conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install transformers==4.19.2 diffusers invisible-watermark
pip install -e .

安装xformers加速
xformers安装包
找到符合自己python，pytorch和cuda版本的xformers安装包

wget https://anaconda.org/xformers/xformers/0.0.16/download/linux-64/xformers-0.0.16-py39_cu11.3_pyt1.12.1.tar.bz2
conda install xformers-0.0.16-py310_cu11.3_pyt1.12.1.tar.bz2
python -m xformers.info

安装成功为下述图片

源码

下载预训练模型权重
文生图
SD2.1-v
Image Inpainting with Stable Diffusion
512-inpainting-ema.ckpt

从Hugging Face下载encoding
当我们运行scripts/txt2img.py时，会发现还需要从hugging face下载encoding，但是国内无法下载，这时候我们会科学上网，同时安装下述依赖，这样就可以下载了

os.environ["http_proxy"] = "http://ip:7890"
os.environ["https_proxy"] = "http://ip:7890"

pip install urllib3==1.25.11
pip install requests==2.27.1

文生图

python scripts/txt2img.py 
--prompt
"a professional photograph of an astronaut riding a horse"
--ckpt
/devdata/chengan/stablediffusion/v2-1_768-ema-pruned.ckpt
--config
configs/stable-diffusion/v2-inference-v.yaml
--H
768
--W
768
--device 
cuda

注意： --device 默认为cpu，要换成 cuda，否则报错

RuntimeError: expected scalar type BFloat16 but found Float

图生图

图生图使用到Gradio依赖，但是会缺少一个文件，使用下述命令补全

wget https://cdn-media.huggingface.co/frpc-gradio-0.2/frpc_linux_amd64
mv frpc_linux_amd64 frpc_linux_amd64_v0.2
mv frpc_linux_amd64_v0.2 /home/chengan/anaconda3/envs/llm/lib/python3.9/site-packages/gradio 

chmod +x /home/chengan/anaconda3/envs/llm/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2

当部署项目到服务器，本地浏览器无法打开Gradio链接，修改inpainting.py文件最后一行

block.launch(server_name="0.0.0.0", server_port=7896, share=True)

python scripts/gradio/inpainting.py configs/stable-diffusion/v2-inpainting-inference.yaml /devdata/chengan/stablediffusion/512-inpainting-ema.ckpt

api

depth2image

pip install -U git+https://github.com/huggingface/transformers.git
pip install diffusers transformers accelerate scipy safetensors

import torch
import requests
from PIL import Image
from diffusers import StableDiffusionDepth2ImgPipeline

pipe = StableDiffusionDepth2ImgPipeline.from_pretrained(
   "stabilityai/stable-diffusion-2-depth",
   torch_dtype=torch.float16,
).to("cuda")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
init_image = Image.open(requests.get(url, stream=True).raw)

prompt = "two tigers"
n_propmt = "bad, deformed, ugly, bad anotomy"
image = pipe(prompt=prompt, image=init_image, negative_prompt=n_propmt, strength=0.7).images[0]

inpaint image

  pipe = StableDiffusionInpaintPipeline.from_pretrained(
      "stabilityai/stable-diffusion-2-inpainting",
      torch_dtype=torch.float16,
  )
  pipe.to("cuda")
  prompt = "two tigers"
  # image and mask_image should be PIL images.
  # The mask structure is white for inpainting and black for keeping as is
  url = "http://images.cocodataset.org/val2017/000000039769.jpg"
  image = Image.open(requests.get(url, stream=True).raw)
  mask_image = np.ones(image.size)
  img = pipe(prompt=prompt, image=image, mask_image=mask_image).images[0]
  plt.imshow(img)
  plt.show()