introduction:
Q: What is oobabooga?
oobabooga-text-generation-webui is a Gradio web user interface for running large language models like Chatglm, RWKV-Raven, Vicuna, MOSS, LLaMA, llama.cpp, GPT-J, Pythia, OPT, and GALACTICA. It aims to be AUTOMATIC1111/stable-diffusion-webui for text generation.
Here is its github link: https://github.com/oobabooga/text-generation-webui
Simply put, it is a language model launcher.
You can load almost all popular models and run them locally. What the language model can do, you can experience what, processing text, personal housekeeper, knowledge consultant, private teaching, this is a weakened version of chatgpt that is not connected to the Internet, but it is different from chatgpt's charging and dialogue review restrictions, customization The local model offers almost unlimited possibilities. (Yes, you can completely train your own cat girl, or experience all kinds of strange content, ahem), and at the same time, you can easily find more than a dozen models on huggingface. The active environment allows small models to develop rapidly , with several new models appearing almost every week, there is always one that meets your requirements.
Q: What are the main functions of oobabooga-text-generation-webui?
The webui supports a lot of functions, most of which I have highlighted in bold.
Similar to stable-diffusion, user-friendly interface.
Friendly one-click installation mode.
The model compatibility is excellent, and all mainstream models can be loaded.
One-click model switching capability, quickly switch between multiple language models.
Self-contained character management, one-click loading
Similar to OpenAI's notepad mode communication interface.
Chat mode for dialogue and roleplay
with Alpaca, Vicuna, Open Assistant, Dolly, Koala, ChatGLM, MOSS, RWKV-Raven, Galactica, StableLM, WizardLM, Baize, Ziya, Chinese-Vicuna, MPT, INCITE, Wizard Mega, Compatible instruction modes for various formats including KoAlpaca, Vigogne, Bactrian, h2o and OpenBuddy
Multimodal pipeline including LLaVA and MiniGPT-4
Markdown output for GALACTICA including LaTeX rendering
Beautiful HTML output for GPT-4chan
Advanced Chat function (send pictures, get audio response with TTS)
very efficient text stream processing
rich parameter presets
LLaMA model supports
4-bitGPTQ model supports
LoRA (loading and training)
llama.cpp model supports
RWKV model supports
8-bit mode
Model layer distribution GPU, CPU and disk
CPU mode
FlexGen
DeepSpeed ZeRO-3
with streaming and non-streaming API
For an introduction to the koishi plugin I made, if you don’t know what koishi is, you can check it here:
https://koishi.chat/zh-CN/
Briefly introduce the oobabooga-testbot plugin:
The oobabooga-testbot plug-in is a qq robot plug-in based on the oobabooga-text-generation-webui language model launcher, which implements some basic instructions, allowing you to communicate with the language model on the qq robot without carrying a computer. It's ok to have a mobile phone. It supports character loading, vits voice reply, AI supplementary tag call plug-in drawing and other functions.
(After all, who doesn't want an online cat girl to accompany them at all times)
Plug-in tutorial post: https://forum.koishi.xyz/t/topic/2391
In fact, this tutorial was published on the koishi forum before. Considering that everyone doesn’t know much about oobabooga, I recreated a tutorial post here to help you install it.
Table of contents:
The first part oobabooga-text-generation-webui
1. oobabooga-text-generation-webui interactive page display.
2. How to build locally (github one-click installation, no error reported during the whole process)
3. How to build locally (github one-click installation, depending on the installation and error reporting)
4. How to build locally (github manual step-by-step installation)
5. Model download
6. General tour of oobabooga interactive page
7. Why I recommend you to use oobabooga-text-generation-webui.
The second part is the use of oobabooga-testbot plug-in and the creation of human design documents
1. Plug-in installation
2. Enable oobabooga API calls
3. Plug-in tutorial
4. Create your own profile
The first part oobabooga-text-generation-webui
1. oobabooga-text-generation-webui interactive page display.
Text exchange page:
text exchange page
Personnel creation page:
Personnel creation page
Model loading and selection interface:
Model loading and selection interface
2. How to build locally (there is no error reported in the whole process)
Two building methods are provided on github:
One-click install package.
Step by step manual installation.
Other ways to build:
Coyude , the leader of station b, provides a lazy package for one-click decompression and installation , but I have not tried whether it is compatible with the plug-in.
Column address:
The boss of station b, AI Skimo, provided a detailed video tutorial.
https://www.bilibili.com/video/BV1fP41127ra/?
This column will focus on using detailed text descriptions to help you use the one-click installation package installation and step-by-step manual construction on github.
GitHub one-click installation process
1. First go to the github homepage. (Scientific Internet access is required, the following content is carried out on the premise of scientific Internet access)
Address: https://github.com/oobabooga/text-generation-webui
2. Find the following options:
You can see that one-click installation now supports Windows, Linux and MacOS.
Let's take the Windows system as an example:
(Note: The following whole process is under the Windows 10 system)
Select the link to download:
The download should be a compressed package:
Unzip
Here are the unzipped files:
Notice! Please be sure to place the file under the full English path, and do not contain spaces or other special symbols
Here is an example: D:\oobabooga\oobabooga_windows
The oobabooga body will take up about 10g of disk space, and the size of a single language model is about 10g, and there are also very large 14b models that can exceed 29g, so please ensure that your disk space is at least about 50g. When loading the model, the data will be imported into the memory, so the faster your disk reading speed, the better. Personally, it is recommended to put it in SSD, which can significantly speed up the loading speed.
pay attention:
The next steps need to go online scientifically in order to install it correctly. If not, please Baidu. I won’t go into details here.
Then we double-click to run start_windows.bat
It will automatically download miniconda and install it in the current directory.
The miniconda installation is complete:
This will let you choose your GPU type
A: NVIDIA, choose A for N card
B: AMD, choose B for card A
C: Apple M Series, Apple chooses C
D: There is no graphics card, or the graphics card is too weak, users who use CPU to run the model choose D (response will be very slow)
My local graphics card is RTX3090 here, so I choose A, enter A, and press Enter to continue.
Then it will start downloading a bunch of stuff automatically, no action required.
This step will take a long time, depending on your internet speed, please wait patiently .
I waited here for about 15 minutes to finish.
This appears to indicate that the download is complete and the decompression installation begins. Please continue to wait.
This instruction appears to start installing dependencies. Please continue to wait .
The probability of error reporting in this step is very high. If there is a red error reporting, it means that there is a problem with your dependent installation. The vast majority of installation problems lie here. See the second method below for details.
The following prompt appears:
Congratulations, you have successfully installed oobabooga.
The current directory structure should look like this:
We run start_windows.bat to test start oobabooga.
Since we have only installed oobabooga and have not downloaded any models, you will be prompted to choose a model to download when starting.
You can choose one to download, but the speed will be slower. My personal suggestion is to use idm to directly go to huggingface to find the corresponding model to download.
Next please skip to the download model section
3. How to build locally (depending on the installation and reporting an error)
It seems that you have encountered an error report unfortunately. The error report may look very different, but the solutions are similar.
Now take this installation error as an example:
This means that the dependencies have not been fully downloaded. The reason for this is generally that your network environment is unstable. If possible, please replace the node and run the installer again. If not, please follow the steps below.
If the installation still fails again:
If it still fails, please close the current console first, and then randomly enter a foreign website to make sure that there is no problem with your ladder , and then proceed to the following steps.
Enter the root directory:
Dependency installation error This problem usually occurs after the installation of miniconda is completed and the github project is successfully git cloned. At this time, your miniconda is generally no problem. Your update directory should be the same as the picture, and there is a file called text-generation-webui folder.
If not, please do not continue and go to the private message or comment area to ask questions.
After making sure the file exists you can proceed:
To fix this we need to enter miniconda first.
Double-click to run the cmd_windows.bat file
A black window will pop up:
Type: cd text-generation-webui
carriage return
Type: pip install -r requirements.txt
carriage return
At this point the program should start installing:
After the installation is complete, oobabooga should be able to run normally.
If something even more unfortunate happens to you, an error is still reported during the installation process, similar to this:
Please continue with the following steps:
Now please go to the change directory, enter text-generation-webui, and find the requirements file:
After opening, delete the three lines of z, save and exit:
Then go back to the root directory, find cmd_windows.bat, double-click to run and create a new console:
Enter and press Enter
pip install peft
pip install transformers
pip install accelerate
cd text-generation-webui
pip install -r requirements.txt
Then wait for the installation to complete.
When you're done, go back to the root directory.
Run update_windows.bat to check all dependencies
If done normally:
Then your oobabooga should be working fine.
Next please skip to the download model section
If the problem has not been solved, please private message up or leave a message in the comment area
4. How to build locally (github manual step-by-step installation)
Next, I will explain the step-by-step manual construction process on github.
Manual installation requires some advanced knowledge, so I assume you already have this knowledge, so the tutorial will be relatively simple.
First you need to install Conda or WSL
Here I will not explain how to install it in detail. There are many tutorials on station b, so you can learn it by yourself.
In the following, I will continue with the windows system and the conda virtual environment installed
Go to the directory where you want to install oobabooga
Edit after mouse selection:
Press Enter, you should open a cmd window:
First create the virtual environment:
conda create -n textgen python=3.10.9
Activate conda:
conda activate textgen
Then according to your system, install Pytorch
The windows environment is the first sentence:
pip install torch torchvision torchaudio
Then install oobabooga
git clone https://github.com/oobabooga/text-generation-webui
cd into the folder
cd text-generation-webui
install dependencies
pip install -r requirements.txt
Wait for the dependencies to run to complete the installation.
For more details, please go to the github page: https://github.com/oobabooga/text-generation-webui
Next please skip to the model download section
If an error occurs
Please try to use the error handling method in the previous method to deal with it
If it still doesn't work, you can choose to ask questions in the comment area, or private message up .
5. Model download
Next I will briefly introduce how to download the model and start it in oobabooga
Let's go to the official website of huggingface: https://huggingface.co/
I'll use the Chatglm model as an example.
Search model:
Pick the first one: https://huggingface.co/THUDM/chatglm-6b
Then we will come to this interface:
Next, there are two options, you can use the downloader that comes with oobabooga to download the model, or choose the file to download yourself.
Use oobabooga built-in downloader
Advantages: Simple, not easy to make mistakes, and the format of the model file must be correct.
Disadvantages: slow.
Method: We first click here:
Then we return to this interface:
Select the L option to Manually specify a Hugging Face model
Then we paste what we just copied into
Press Enter and it will start downloading automatically.
The download directory is: \oobabooga_windows\text-generation-webui\models
Inside the folder is the downloaded model.
This way only occurs when you don't have any models
If you already have a model and still want to use oobabooga to download the model, you can jump to the introduction of the model tab of the general navigation on the interactive page.
Download the model file yourself:
Advantages: fast
Cons: manual
Method: We first enter the model storage directory
\oobabooga_windows\text-generation-webui\models
Create a new folder, say:
Then we go to Chatglm's huggingface page:
https://huggingface.co/THUDM/chatglm-6b
Select files:
Download all the files here into the file you just created:
The inside of the completed model folder should look like this:
Notice!
Different types of models have different model formats. Generally speaking, you only need to download all the files in huggingface. If you still have questions, please join the group to ask.
Next we continue to load the model of oobabooga
You should have downloaded the model.
If there is only one model in the model folder, oobabooga will load it automatically . If there are multiple models, like me, it will jump out of the model selection page:
Now we take Vicuna-13b as a case to start the model.
(Why not use Chatglm as an example because Chatglm needs to install icetk additionally, I will talk about it in detail later)
(Attention! Please turn off clash, oobabooga and clash will conflict, causing the page to start but unable to talk to the model)
pop up
Running on local URL: http://127.0.0.1:7860
To create a public link, set share=True in launch().
It means that the startup is successful, and then, like stable-diffusion, we visit 127.0.0.1:7860
You can see the webui interface:
Use the Chatglm model in the oobabooga above
When we load the Chatglm model again, we will encounter an error of missing dependencies, generally speaking, it is icetk:
(Couldn't find a corresponding picture yet)
To fix this, we need to install icetk.
First we come to the oobabooga folder:
Select cmd_windows.bat to run
Type pip install icetk
Enter to run
Wait for the installation to complete (if an error is reported, it may be that you did not open Science Internet)
Then go back to the oobabooga folder and find the webui.py file:
Right click and open with Notepad:
Scroll down to around line 140-180 and find "def run_model()"
behind python sever.py
Add --trust-remote-code
save and exit
Then reopen oobabooga and you should be able to load the Chatglm model.
If you encounter more questions, please ask questions in the comment area, or private message up.
6. General tour of oobabooga interactive page
I'll briefly describe oobabooga's interface options and how to talk to and switch models.
At the top of the page, we can see these tabs.
The first is the main interface where we talk to the model
The second is the character design/role loading and setting interface
The third is the detailed parameter adjustment of the model
The fourth is the model switching, loading and downloading interface
The fifth is the lora training interface
The sixth is the webui tab
I will mainly introduce the first main interface for talking to the model and the fourth model switching, loading and downloading interface
I won't talk about the detailed settings, just use the translator to look at it yourself.
Text generation interface
We enter the content we want to talk to the model in the input, and then use the yellow generate button to talk to the model. The following buttons are some normal buttons for chatting, and you can understand them after you explore them yourself.
Then there is this part. Here you have three modes, which basically switch between different dialog modes. In the chat mode shown here, after selecting a persona, a corresponding avatar will appear, which increases the sense of substitution:
Then the following chat style
This option depends on what kind of chat style your model supports. Generally speaking, oobabooga will automatically choose the best one for you.
The bottom is the gallery
This is used to load the character. oobabooga comes with an example role:
Select the character card to load it.
That's all for the introduction of the main interface for talking to the model.
Next is the fourth model switching, loading and downloading interface
model interface
We select the tab to come to the model:
I will partition to explain in detail:
The upper left corner is the model selection interface, we can directly select the model to load, the right is the lora interface, we can also choose to load, the following is the option of the Transformers model, you can choose to limit the memory size occupied by the model, and you can choose the mode to use on the right:
Here are the options used by the GPTQ model. Generally speaking, oobabooga will automatically select the appropriate one, but when using a newer model, you may have to select the corresponding mode yourself:
This part is whether to automatically load the model, that is, whether it will be automatically loaded after you select the model in the above option.
The following is the automatic download of the model, which is the same as the downloader that comes with oobabooga mentioned earlier.
Just enter the name of huggingface in the box below, and click download to automatically download the model:
Here are the tabs for the llama model:
Here are the buttons to apply lora, load the model, reload the model, and save the model settings:
This is the end of the introduction to the oobabooga interface
In addition to the parts I introduced, oobabooga also provides a lot of various types of functions. At the same time, like stable-diffusion, it also supports plug-in installation, and you can explore by yourself.
You can also leave a message in the comment area or private message up to discuss.
7. Why I recommend you to use oobabooga-text-generation-webui
This part is mainly my subjective opinion, everyone just treat it as Amway.
I am personally very interested in language models (mainly because I want a personal assistant). Since openai released chatgpt, I have started to pay attention to small models extensively. However, the various models in the early stage have too high a threshold for people like me who are not very good at programming, and I can't run many models.
While paying a lot of attention to small models, I noticed that foreign small models are developing very fast. From RWKV, alpaca, Vicuna, to wizard, stableVicuna, the iteration speed of small models is very fast, almost every few days. With the emergence of a new model, I really hope that more people can participate in this change, and oobabooga-text-generation-webui is a very good starting point.
As a language model launcher, oobabooga-text-generation-webui can be said to have the widest range of support for models. At the same time, the efficient model loading method largely meets my needs for a large number of model tests. At the same time, the built-in character setting loading function also satisfies my personalization needs.
If you, like me, have a strong interest in small models, and hope to be able to play the model as soon as it is released, then I strongly recommend you to use it .
The second part of plug-in use and creation of human document
1. Plug-in installation
Search oobabooga in koishi's plugin market
Click Add to install
It can be used normally after configuration on the plug-in configuration page:
2. oobabooga enables api service
Next, I will explain how to open the api option of oobabooga, so that the plug-in can call the local oobabooga.
We come to the oobabooga folder and select webui.py
Find lines 140-180 down
Find "def run_model()"
behind python sever.py
add --api
Then save and close, start the webui normally
You can see in the Interface mode tab:
In this way, the api service will start normally, and your plug-in should be able to call oobabooga through the default url address.
2. Plug-in tutorial
As of May 24, the plugin version number is 1.2.0
Plugins provide a variety of commands:
Let's take a look at the configuration page first:
apiURL is used to fill in the api address of your oobabooga. If you want to open the api of oobabooga, please see the upper floor.
The user logic is as follows:
First, let's try the oob.list command to view the current character setting
Then we use the oob.load command to load the character set we want to use
Inside the plugin folder, a new history will be created
Then we can use oob to talk to the model
The oob command is a basic command, by using the oob command you can directly talk to the model
The history will be saved in the corresponding file in real time, and the historylimit will limit the length of the context, depending on your settings.
Using the oob.undo command, you can withdraw a question just now
Use the oob.reset command to reset the history of the current session
Use the oob.del command to delete the current session to load a new avatar.
The oob.tag command can be used to let Ai help to add tags and draw.
The oob.check command is a check command that checks whether a session already exists
The current plugin already supports nicknames and @wake
4. Create your own profile
Please note that the character document is completely read and saved locally, and it will be cleared every time the plugin is updated, please keep a backup for yourself.
first we come to
koishi-plugin-oobabooga-testbot\lib\characters
under the folder
Use Notepad to open any profile file, you will find that the format of profile files is similar.
The persona file consists of two parts:
Basic character design + example dialogue
Personality files are actually stored in the form of arrays, here is an example that is easier to understand:
[“Hello”,“Hello”]
This forms a set of conversations.
But our character setting file is a little more complicated, as you can see, we first gave a character setting:
"You are an artificial intelligence language model, you will reply in Chinese, and you will act as an assistant, giving a helpful and friendly reply. You only need to reply as an assistant."
The beginning and end are then added to the standard dialogue example, here is an example:
[“You: ||Hello||”,“Assistant: ||Hello||”]
This is to allow the model to better recognize which are human settings and which are example dialogues. At the same time, standardize the return format of the model to facilitate our subsequent processing.
Note the name of your profile file:
This is the inside of the Aqua character set, and you can see that the conversation format here has changed to:
[“You: ||Hello||”,“Aqua: ||Hello||”]
This is because the plug-in will automatically recognize the name of the character file and let the model play the role, which can make the model's reply more in line with the character.
Now that you have a general understanding of the format of the avatar file, you can try to create your own avatar.
For further discussion, please private message up or discuss in the comment area
something to say
It would be amazing if you can stick to this point, you can endure my long and joyless tutorial. I'm so tired, writing a thesis, plug-ins, and tutorials at the same time, I feel my body is hollowed out. But if you can learn something from my tutorial, or even get a little inspiration, I am very happy. I hope you can use oobabooga-text-generation-webui happily. Now that the language model is developing rapidly, you can keep up with the pace of the times and experience the convenience brought by the small model together.
——smzh Sincerely