foreword
Recently, AIGC (that is, AI Generated Content , which refers to the use of artificial intelligence technology to generate content) has really exploded into the sky. In addition to chatGPT , which has been squeezed to the full load of the server , another field that has attracted much attention is undoubtedly AI painting. Using an open source class of "diffusion " models, you can paint with AI at any time. In addition to the common web version registration->enter text->call API->wait for the painting to be generated->take a screenshot or download and save the process, of course, as a developer who does not want to be restricted, I believe there must be some friends who want to use it on their own computers. , Unimpeded creation. Then follow our three simple steps below to see how you can easily and quickly build an environment for running the Stable Diffusion model pipeline on your own computer, and let AI help you draw as you like.
1. OpenVINO ™ optimizes and accelerates Stable Diffusion inference deployment
Since the model pipeline of Stable Diffusion contains several deep learning models of large size, it has certain requirements for hardware computing resources for running reasoning. Here we use OpenVINO to convert the Stable Diffusion model in the original PyTorch format, so as to optimize the model pipeline and accelerate inference.
How to do it? We open source all the codes in the OpenVINO Notebooks repository, for details, please refer to ( openvino_notebooks/notebooks/225-stable-diffusion-text-to-image at main openvinotoolkit/openvino_notebooks GitHub ). In this code example, we take the Stable Diffusion model and convert the model to the OpenVINO Intermediate Representation (IR) format so that it can run efficiently on Intel® GPUs . Also, by compressing the FP32 model to FP16 , we cut the model size in half (nearly in half), and it now requires a lot less RAM/VRAM to run. Most importantly, due to the addition of Intel® Xe Matrix Extensions ( XMX ), GPU processing speed has also been significantly improved.
Here are some interesting results I got from running this notebook . With the Intel Sharp ™ A770m discrete graphics, I can achieve about 6.0 iterations per second (without using debug mode). This means it typically takes less than 10 seconds to generate one of the high-quality images below.
2. Installation steps
If you need to install OpenVINO Notebooks , the detailed installation steps can refer to here (for Windows ): Windows · openvinotoolkit/openvino_notebooks Wiki · GitHub
If you are a Linux user, you can click this link: https://github.com/openvinotoolkit/openvino_notebooks/wiki/Ubuntu
1. Install Python 3.10.x. ( and Git, C++ Redistributable (if using Python 3.8) ) and create a virtual environment
python3 -m venv openvino_env
openvino_env\Scripts\activate2. Implement Git clone for the directory
git clone --depth=1 https://github.com/openvinotoolkit/openvino_notebooks.git
cd openvino_notebooks
3. Install all libraries and dependencies
python -m pip install --upgrade pip wheel setuptools
pip install -r requirements.txt3. Run Notebook and perform AI painting
Load all notebooks with the following command:
jupyter lab notebooksIn the 225-stable-diffusion-text-to-image folder, you can find the Notebook code example for AI painting:
Run all cells:
Wait for a while to see the AI-generated paintings~
Take a look at my generated paintings.
4. Running tips and extending to more models
Looking at the code in the Notebook, we actually optimized the PyTorch pipeline and used OpenVINO to execute the code to accelerate reasoning and drawing generation.
1. Download the OpenVINO IR model directly:
The first download of the Stable Diffusion PyTorch model and conversion may take a little while. After completion, you will get a set of model files in OpenVINO IR format. For convenience, we have updated these pre-trained optimization models to huggingface ( https://huggingface.co/bes-dev/stable-diffusion-v1-4-openvino ), and you can download and use them directly.
2. Select the device to run inference on:
OpenVINO can easily help developers deploy models on different hardware platforms to run inference. Now, if your device has an Intel Sharp ™ discrete graphics card in addition to the CPU, you can change the code of the device name in the figure below to "GPU" . By default it uses " AUTO " and will automatically switch to the detected GPU .
let it run on the GPU
Automatic plugin. It uses the CPU first, then automatically switches to the GPU.
3. Adjust the total number of steps to generate the painting, and improve the input text prompt:
In this step, I set the step to 30 . Ideally, I'd use 50 , which gives the best looking results.
By modifying the input text, you can generate different scenarios here. For really cool images, you can try some helpful tips put together by the community. Best 100+ Stable Diffusion Prompts: The Most Beautiful AI Text-to-Image Prompts | Metaverse Post
4. Use the Chinese version of the model to carry out national style painting:
In addition to the Stable Diffusion model shown in the Notebook, the above Notebook code example can be easily extended to more models after a few lines of code modification, such as the Chinese version of the Stable Diffusion "Taiyi" model IDEA-CCNL/Taiyi- Stable- Diffusion-1B-Anime-Chinese-v0.1 Hugging Face
Change the name of the pre-trained model here to "IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Anime-Chinese-v0.1"
And modify the tokenizer here accordingly
change into
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1")Then enter some Chinese prompt words, and you can create the national style painting
Summarize
At the moment, if you want to understand how " Stable Diffusion " works, and how Intel hardware accelerates it, OpenVINO Notebooks is undoubtedly the first choice. If you have any questions or want to showcase some of your best work, please leave a comment here or via our GitHub discussion board ! Happy coding, everyone.
openvinotoolkit/openvino_notebooks · Discussions · GitHub · GitHub