Personal website: https://tianfeng.space
Preface
Since the announcement of LCM, the update speed of related applications has been nb this week, and various real-time painting workflows have emerged. Previously, they could only be used by krea internal beta qualifications. Let us take a look at what happened last week. Bar!
Netdisk (the real-time painting workflow is downloaded from Hotpot): https://pan.quark.cn/s/223f12a85fa0
event flow
This explanation is based on comfyui. If it is not installed, you can take a look. The model file can be placed in webui, because my comfyui and webui share a model folder.
comfyui installation: https://blog.csdn.net/weixin_62403633/article/details/133994238?spm=1001.2014.3001.5501
A brief introduction to LCM: https://blog.csdn.net/weixin_62403633/article/details/134517970?spm=1001.2014.3001.5501
Regarding the issue of workflow construction for novices: You can look at other people’s work first. If you use it more, I believe you can build it yourself later.
Reference website: https://comfyworkflows.com/
Reference website: https://civitai.com/search/models?sortBy=models_v5&query=workflow
Reference website: https://openart.ai/workflows/dev?sort=latest
1.LCM real-time painting workflow
I will announce my nodes first,
Which installation is missing to prevent the workflow from being reproduced and restarting?
After installation, remember to update everything
Update the one you just installed. Click on the two script files.
Solution A: ComfyUI_Custom_Nodes_AlekPet node (built-in artboard)
Drag into the workflow, the ComfyUI_Custom_Nodes_AlekPet node has a drawing board node, you can draw directly and then output the results. Re-select the model and lcm-lora model on the node, check automatic sequence generation on the right, click the prompt word queue, and start real-time drawing
Option B: comfyui-mixlab-nodes nodes (screencasting)
It’s the same as the previous workflow, except that the artboard node is replaced by a screen projection node.
Choose screencast ps
Return to comfyui, select the ps drawing board area with a rectangular shape (just drag the upper left corner to the lower right corner)
Finally, click liveRun to turn on picture-in-picture. Click the prompt word queue in comfyui to draw in real time.
Assuming that PS is selected for screencasting, real-time painting will be performed based on PS painting at this time, and next to it is a picture-in-picture real-time preview.
Solution C: krita painting tool combined with comfyui
https://github.com/Acly/krita-ai-diffusion#installation
Downloaded all applications and plug-ins
https://github.com/Acly/krita-ai-diffusion/releases/download/v1.8.2/krita_ai_diffusion-1.8.2.zip
Click to install the application directly. After opening, place the plug-in in this directory (if the plug-in is updated, just download and replace it)
Open krita, in the settings options, configure python plug-in management in krita, check AI Image Diffusion and restart
Open comfyui and krita, and create a new canvas. Generally, according to the model sd1.5, it is 512x512, and sdxl is 1024x1024. After the creation is completed, the following picture will appear in the lower right corner (if not, set->panel list->check AI Image Diffusion)
Click configure. The red color indicates that the node is missing. You can directly click to download and unzip it and put it in the comfyui node directory. You can also go to comfyui to search and install. If there is no git clone or download and unzip it,
Meet the following conditions at the same time
https://github.com/Acly/krita-ai-diffusion/blob/main/doc/comfy-requirements.md
In addition to downloading the missing nodes, the clip_vision model also needs to be placed in the models/clip_vision/SD1.5 directory.
NMKD enlarges the model to the models/upscale_models directory
Put the ipadapter model into custom_nodes/ComfyUI_IPAdapter_plus/models (directly copy the one under weui)
This controlnet model is placed in the node, so the shared model cannot be recognized; lcm-lora is also copied and placed in the lora directory.
(Renamed lcm-lora-sdv1-5.safetensors lcm-lora-sdxl.safetensors)
Okay, you're basically done, restart.
You can create new vector layers, pose images, depth images, etc. You can study the rest! If you have any questions, send a private message.
2.SDXL-turbo
https://huggingface.co/stabilityai/sd-turbo/tree/main
SD-Turbo is a refined version of Stable Diffusion 2.1, trained with real-time synthesis. SD-Turbo is based on a novel training method called Adversarial Diffusion Distillation (ADD) (see technical report), which allows sampling of large-scale base image diffusion models at high image quality in 1 to 4 steps. This approach uses fractional distillation to leverage large-scale off-the-shelf image diffusion models as the teacher signal and combines it with an adversarial loss to ensure high image preservation even in low-step regimes of one or two sampling steps. Trueness.
After downloading the model and placing it, I built a workflow experience. The objects were passable and the character effects were average. I look forward to subsequent optimization.
Two workflows are provided, and the corresponding model can be downloaded at https://civitai.com/models/219765/sdxl-turbo-lcmworkflow
2.SVD
SVD: The model is trained to generate 14 frames at a resolution of 576x1024 given the same size context frame. SVD-XT: Same architecture as SVD but fine-tuned for 25 frame generation.
SVD：https://huggingface.co/stabilityai/stable-video-diffusion-img2vid
SVD-XT：https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt
Current limitations
- The resulting video is quite short (<=4 seconds) and the model does not achieve perfect photorealism.
- The model may produce videos with no motion or very slow camera pans.
- Models cannot be controlled via text.
- The model cannot render legible text.
- In general, faces and characters may not be generated correctly.
- The autoencoding part of the model is lossy.
Just download the corresponding model file and use it. The network disk provides some workflows. For more information, please go to the website I provide to download it!
