I. Introduction
Through the previous introduction to the large language model, we learned that, in fact, some mainstream large language models that have been released, such as GPT-3.5, GPT-4 released by OpenAI, and Bard and PaLM 2 multimodal models released by Google , Claude 2 recently launched by Anthropic, or ChatGLM 2 released by Tsinghua University in China and Zhipu AI, etc. are all general-purpose LLM models, which are based on massive Internet data for training and reasoning. Because it involves a wide range of fields, you will It is found that these models are more or less unfamiliar with some knowledge or answering wrongly, and a lot of confidential data cannot be obtained, so to a certain extent, although most of the problems can be solved. However, for some specific fields, the existing model capabilities still have certain limitations. At this time, we need to effectively supplement and fine-tune the knowledge that the model has not absorbed to release the model's capabilities, so that the model can better Adapt to the nuances of a specific use case or domain, resulting in more accurate results.
In addition to model fine-tuning, we also introduced the application case of using LangChain combined with LLM to build a custom knowledge base. The semantic understanding of the language model and the retrieval of the vector retrieval results can also solve the purpose of expanding the model knowledge base to solve some practical problems of the enterprise. Compared with model fine-tuning training, this solution is very low in cost. Therefore, due to the popularity of the GPT model and the rigid demand for building a knowledge base based on GPT, the vector database was once pushed to a climax. Various top AI companies and Unicorns have launched their own vector databases, and for applications developed based on PostgreSQL, they also provide lightweight solutions for vector retrieval. Even MySQL seems to be less popular.
It is not because of the vector similarity retrieval technology and the high cost of model fine-tuning that this solution is meaningless. In fact, it is not the case. If you really want to launch an AI product with your own core competitiveness in a specific field, the most suitable way is still It is still based on some LLMs that are currently open source or closed source to fine-tune their own business data to train smarter and more humanized model products. Some time ago we introduced a fine-tuning solution for local open source models "Open Source LLM Fine-tuning Training Guide" : How to build your own LLM model", today we mainly introduce the fine-tuning based on OpenAI's GPT model to build your own personal model.
2. What is model fine-tuning?
Before specifically introducing how to fine-tune the model, we need to understand that fine-tuning the model and model development and training are completely different concepts. In many papers published on large language models, the dataset size, training environment, and inference time for model training will be introduced. These are the time and cost of the complete process from 0 training inference to evaluation and maturity of the model. It is almost difficult for ordinary individuals or companies to afford this kind of cost investment, and model fine-tuning actually refers to the model based on the already trained model . Above , through the process of further training GPT-3 on a custom use-case dataset. Often used to improve the performance of a model, which allows the model to better adapt to the nuances of a specific use case or domain, resulting in more accurate results.
If you need to fine-tune the GPT model, first you need to understand that the GPT model is a natural language processing model for generating text. It is pre-trained on a large-scale corpus and learns the structure and regularity of language. However, since each task or dataset has its own characteristics and requirements, it is usually necessary to fine-tune the GPT model to make it more suitable for your specific task.
The steps to fine-tune a GPT model usually include the following steps:
- Prepare the dataset: First, you need to prepare a dataset related to your task. This dataset should include the input text and the corresponding target output. For example, if your task is to generate movie reviews, then your dataset should include movie reviews and corresponding sentiment labels.
- Adjust the model structure: The GPT model usually consists of a multi-layer Transformer network. You can fine-tune the structure of the model according to the needs of the task. For example, adding some task-specific layers or adjusting the number of layers of the model.
- Reset model weights: Initialize the weights of the GPT model to pre-trained weights, and freeze these weights. This is done to preserve what the model has learned during the pre-training phase and to prevent over-tuning the model during fine-tuning.
- Fine-tuning the model: Using the prepared data set, update the weight of the model through the backpropagation algorithm. During fine-tuning, you can tune hyperparameters like learning rate, batch size, etc. to optimize the performance of your model.
- Evaluation and optimization: During the fine-tuning process, you can periodically use the validation set to evaluate the performance of the model. According to the evaluation results, you can further optimize the model, such as adjusting hyperparameters, increasing training rounds, etc.
Through such a fine-tuning process, you can better adapt the GPT model to your task and achieve better performance. It should be noted that the fine-tuning process requires a lot of computing resources and time, so the cost of time and resources needs to be carefully weighed in actual operation.
3. Which GPT models can be fine-tuned?
GPT models that can be fine-tuned include Ada, Babbage, Curie, and Davinci. These models belong to the GPT-3 family. Also, it's worth noting that fine-tuning doesn't currently work on newer GPT-3.5-turbo models or other GPT-4s .
According to the latest news released by OpenAI, it is planned to close the Ada, Babbage, Curie and Davinci models on January 4, 2024, and the launch time of fine-tuning based on the latest GPT-3.5 Turbo and GPT-4 is not clear, and it is expected to be in the second half of this year release. If you need it urgently in the short term or if you want to learn something, you can pay attention to it, and if you are not in a hurry, you can wait for the fine-tuned version of the new model to be released before considering it.
On July 6, 2023, we announced the deprecation of the ada, babbage, curie, and davinci models. These models (including fine-tuned versions) will be closed on January 4, 2024. We are actively working on fine-tuning the upgraded base GPT-3 model as well as GPT-3.5 Turbo and GPT-4, and we recommend waiting for these new options to become available rather than fine-tuning based on the soon-to-be-deprecated model.
4. Which application scenarios are suitable for fine-tuning?
For Fine-Tuning GPT, there are several good application scenarios: text classification and conditional generation.
4.1. Text Classification
For classification problems, each input is assigned to one of the predefined categories, here are some examples:
- Ensuring truthful representations: If a company wants to verify that the ads on its website mention the correct products and companies, it can Fine-Tuning the classifier to filter out incorrect ads and ensure the model doesn’t make things up.
- Sentiment Analysis: This involves classifying text based on sentiment, such as positive, negative, or neutral.
- Email Classification: Segregate incoming emails into one of several predefined categories which can be converted to numbers, works well for up to ~500 categories.
4.2, condition generation
This class of problems involves generating content from a given input. Applications include rewriting, summarization, entity extraction, product description writing, virtual assistants (chatbots), and more. Here are some examples:
- Create compelling ads from Wikipedia articles. In this generative use case, make sure the provided samples are of high quality, as the Fine-Tuned model will try to mimic the style (and errors) of the samples.
- Entity extraction. This task is similar to the language switching problem. Performance can be improved by sorting the extracted entities alphabetically or in the order they appear in the original text.
- Customer support chatbots. Chatbots typically contain relevant context about the conversation (order details), a summary of the conversation, and recent messages.
- Product description based on technical attributes. Convert input data into natural language to achieve better performance in this environment.
5. How to fine-tune the GPT model?
Model fine-tuning is because models such as ChatGPT and GPT-4 are not all-powerful AI, and their answers in many vertical fields may be wrong. One of the main reasons is the lack of domain-specific training data. If we have a wealth of domain-specific data, we can use this data to "fine-tune" the model to perform well in that domain. After fine-tuning, we can directly ask the model questions, instead of using embeddings to query relevant materials as before, and then submit the found materials to OpenAI for answers.
OpenAI's model fine-tuning process is not complicated. You only need to provide data to OpenAI, and the fine-tuning process will be performed in a "black box" in the cloud. The data format to be provided is a text file, each line is a Prompt, and the content corresponding to the Completion interface generated by the Prompt.
This article will introduce the process of fine-tuning the GPT-3 model on the user's own data using Python, covering all steps from obtaining API credentials to preparing data, training the model, and validating the model.
5.1. Dataset Standards
We will fine-tune the GPT-3 model for the question answering scenario, which contains structured question answering patterns designed to help the model understand the tasks the model needs to perform. Each pair of questions and answers is kept in a consistent format throughout the training and testing data.
Your data must be a JSONL document where each line is a hint-completion pair corresponding to a training example. Data can be easily converted to this file format using the CLI data preparation tool provided by OpenAI.
The more training examples the better. According to the official recommendation there are at least a few hundred examples. In general, each doubling of the dataset size leads to a linear increase in model quality.
Instances of the question answering dataset follow the format:
{
"prompt": "my prompt text ->",
"completion": "the answer of the prompt. \n"
}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
...- "prompt": is the input text read and processed by the model. The primary delimiter is the arrow symbol (->), used to distinguish a prompt from an expected response.
- "completion": is the expected response to the prompt. The backslash "\n" symbol is used as a stop sequence to indicate the end of the answer.
After understanding the dataset format, we can generate training and validation datasets as shown below. These prompts and completions are generated using ChatGPT.
training_data = [
{
"prompt": "中国的首都是哪里？->",
"completion": """中国的首都是北京。\n"""
},
{
"prompt": "中国的最长河流是哪条？->",
"completion": """中国的最长河流是长江。\n"""
},
{
"prompt": "中国的最高山峰是哪座？->",
"completion": """中国的最高山峰是珠穆朗玛峰。\n"""
},
{
"prompt": "中国的国花是什么？->",
"completion": """中国的国花是牡丹。\n"""
},
{
"prompt": "中国的四大发明包括哪些？->",
"completion": """中国的四大发明包括指南针、造纸术、火药和印刷术。\n"""
},
{
"prompt": "中国的传统节日春节是在哪个月份？->",
"completion": """中国的传统节日春节是在农历正月。\n"""
},
{
"prompt": "中国的长城是世界上最长的人造建筑物吗？->",
"completion": """是的，中国的长城是世界上最长的人造建筑物。\n"""
},
{
"prompt": "中国的国旗由几颗星组成？->",
"completion": """中国的国旗由五颗星组成。\n"""
},
{
"prompt": "中国的国歌是什么？->",
"completion": """中国的国歌是《义勇军进行曲》。\n"""
},
{
"prompt": "中国的面积排名世界第几？->",
"completion": """中国的面积排名世界第三。\n"""
}
]
validation_data = [
{
"prompt": "中国的第一位主席是谁？->",
"completion": """中国的第一位主席是毛泽东。\n"""
},
{
"prompt": "中国的最长高速公路是哪条？->",
"completion": """中国的最长高速公路是京藏高速公路。\n"""
},
{
"prompt": "中国的最大湖泊是哪个？->",
"completion": """中国的最大湖泊是青海湖。\n"""
},
{
"prompt": "中国的省级行政区划有多少个？->",
"completion": """中国的省级行政区划共有34个。\n"""
},
{
"prompt": "中国的古代丝绸之路起点是哪里？->",
"completion": """中国的古代丝绸之路起点是西安。\n"""
},
{
"prompt": "中国的十二生肖中有哪些动物？->",
"completion": """中国的十二生肖包括鼠、牛、虎、兔、龙、蛇、马、羊、猴、鸡、狗和猪。\n"""
},
{
"prompt": "中国的国家体育馆被称为什么？->",
"completion": """中国的国家体育馆被称为鸟巢。\n"""
},
{
"prompt": "中国的国家图书馆位于哪个城市？->",
"completion": """中国的国家图书馆位于北京。\n"""
}
]5.2. Prepare the basic environment
We need to prepare a working environment and recommend using the OpenAI Command Line Interface (CLI). To install it, run
pip install --upgrade openai(The instructions below are for version 0.9.4 and later. Also, OpenAI CLI requires python 3.)
The OPENAI_API_KEY environment variable is set by adding the following line to a shell initialization script (eg.
export OPENAI_API_KEY="<OPENAI_API_KEY>"5.3, prepare the data set
As shown above, dealing with list formats can be convenient for small datasets. However, saving data in JSONL (JSON Lines) format has several benefits. Benefits include scalability, interoperability, simplicity, and compatibility with the OpenAI API, which requires data in JSONL format when creating fine-tuning jobs.
The following code utilizes the helper function prepare_data to create training and validation data in JSONL format:
import json
training_file_name = "training_data.jsonl"
validation_file_name = "validation_data.jsonl"
def prepare_data(dictionary_data, final_file_name):
with open(final_file_name, 'w') as outfile:
for entry in dictionary_data:
json.dump(entry, outfile)
outfile.write('\n')
prepare_data(training_data, "training_data.jsonl")
prepare_data(validation_data, "validation_data.jsonl")In a Google Colab notebook, dataset preparation can be done using the following statements for training and validation data.
!openai tools fine_tunes.prepare_data -f "training_data.jsonl"
!openai tools fine_tunes.prepare_data -f "validation_data.jsonl"Analyzing...
- Your file contains 22 prompt-completion pairs. In general, we recommend having at least a few hundred examples. We've found that performance tends to linearly increase for every doubling of the number of examples
- More than a third of your `prompt` column/key is uppercase. Uppercase prompts tends to perform worse than a mixture of case encountered in normal language. We recommend to lower case the data if that makes sense in your domain. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more details
- More than a third of your `completion` column/key is uppercase. Uppercase completions tends to perform worse than a mixture of case encountered in normal language. We recommend to lower case the data if that makes sense in your domain. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more details
- All prompts end with suffix `？ ->`
- All prompts start with prefix `《`
- All completions end with suffix `\n`
- The completion should start with a whitespace character (` `). This tends to produce better results due to the tokenization we use. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more details
Based on the analysis we will perform the following actions:
- [Recommended] Lowercase all your data in column/key `prompt` [Y/n]: Y
- [Recommended] Lowercase all your data in column/key `completion` [Y/n]: Y
- [Recommended] Add a whitespace character to the beginning of the completion [Y/n]: Y
Your data will be written to a new JSONL file. Proceed [Y/n]: Y
Wrote modified file to `training_data_prepared.jsonl`
Feel free to take a look!
Now use that file when fine-tuning:
> openai api fine_tunes.create -t "training_data_prepared.jsonl"
After you’ve fine-tuned a model, remember that your prompt has to end with the indicator string `？ ->` for the model to start generating completions, rather than continuing with the prompt. Make sure to include `stop=["\n"]` so that the generated texts ends at the expected place.
Once your model starts training, it'll approximately take 2.75 minutes to train a `curie` model, and less for `ada` and `babbage`. Queue will approximately take half an hour per job ahead of you.Finally, we upload the two datasets to the OpenAI developer account as follows:
training_file_id = upload_data_to_OpenAI(training_file_name)
validation_file_id = upload_data_to_OpenAI(validation_file_name)
print(f"Training File ID: {training_file_id}")
print(f"Validation File ID: {validation_file_id}")After the training data is uploaded successfully, a unique file ID and file-related information will be returned.
<File file id=file-hKjjDPr885IqmsfcDJvEsidl at 0x7f17f5725f80> JSON: {
"object": "file",
"id": "file-hKjjDPr885IqmsfcDJvEsidl",
"purpose": "fine-tune",
"filename": "file",
"bytes": 15103,
"created_at": 1690133676,
"status": "uploaded",
"status_details": null
}5.4. Create a fine-tuning model
In order to fine-tune, we need to follow the following two steps: (1) define the hyperparameters, (2) start the fine-tuning process.
Every fine-tuning job starts with a base model, the Curie model by default. The choice of model directly affects the performance of the model and the cost of running the fine-tuned model. The model that can be selected for fine-tuning can be one of: ada, babbage, curie, or davinci.
Once you start fine-tuning work, it may take some time to complete. The created fine-tuning task needs to be queued, and training our model may take minutes or hours, depending on the model and dataset size.
Next we will fine-tune the daVinci model using the training and validation datasets. We will set batch_size to 3 and learning_rate_multiplier to 0.3 and run fine-tuning in 15 epochs.
create_args = {
"training_file": training_file_id,
"validation_file": validation_file_id,
"model": "davinci",
"n_epochs": 4,
"batch_size": 3,
"learning_rate_multiplier": 0.3
}
response = openai.FineTune.create(**create_args)
job_id = response["id"]
status = response["status"]
print(f'Fine-tunning model with jobID: {job_id}.')
print(f"Training Response: {response}")
print(f"Training Status: {status}")The above code generates the following information for jobID ( ft-ByTj1ZNnSn6Fl7rvm8MhzcVV ), training response and training status ( pending ).
<FineTune fine-tune id=ft-ByTj1ZNnSn6Fl7rvm8MhzcVV at 0x7f17f576d760> JSON: {
"object": "fine-tune",
"id": "ft-ByTj1ZNnSn6Fl7rvm8MhzcVV",
"hyperparams": {
"n_epochs": 4,
"batch_size": 3,
"prompt_loss_weight": 0.01,
"learning_rate_multiplier": 0.3
},
"organization_id": "org-MplDvRDOWGVmqFOhYs8cLCMd",
"model": "davinci",
"training_files": [
{
"object": "file",
"id": "file-hKjjDPr885IqmsfcDJvEsidl",
"purpose": "fine-tune",
"filename": "file",
"bytes": 15103,
"created_at": 1690133676,
"status": "processed",
"status_details": null
}
],
"validation_files": [],
"result_files": [],
"created_at": 1690133686,
"updated_at": 1690133686,
"status": "pending",
"fine_tuned_model": null,
"events": [
{
"object": "fine-tune-event",
"level": "info",
"message": "Created fine-tune: ft-ByTj1ZNnSn6Fl7rvm8MhzcVV",
"created_at": 1690133686
}
]
}The fine-tuning task does not output any relevant information in the pedding state. But we can gain a deeper understanding of the training process by running the following code:
import signal
import datetime
def signal_handler(sig, frame):
status = openai.FineTune.retrieve(job_id).status
print(f"Stream interrupted. Job is still {status}.")
return
print(f'Streaming events for the fine-tuning job: {job_id}')
signal.signal(signal.SIGINT, signal_handler)
events = openai.FineTune.stream_events(job_id)
try:
for event in events:
print(f'{datetime.datetime.fromtimestamp(event["created_at"])} {event["message"]}')
except Exception:
print("Stream interrupted (client disconnected).")All epochs are generated below, along with the status of fine-tuning, fine-tuning was successful.
Streaming events for the fine-tuning job:ft-ByTj1ZNnSn6Fl7rvm8MhzcVV
2023-07-23 15:11:54 Created fine-tune: ft-ByTj1ZNnSn6Fl7rvm8MhzcVV
2023-07-23 15:12:30 Fine-tne costs $0.12
2023-07-23 15:12:30 Fine-tune enqueued.Queue number:0
2023-07-23 15:12:31 Fine-tunestarted
2023-07-23 15:14:38 Completedepoch 1/4
2023-07-23 15:14:41 Completedepoch 2/4
2023-07-23 15:14:45 Completedepoch 3/4
2023-07-23 15:14:49 Completedepoch 4/4
2023-07-23 15:16:07 Uploaded model:davinci:ft-personal-2023-07-23-20-16-06
2023-07-23 15:16:08 Uploaded result file:file-hKjjDPr885IqmsfcDJvEsidl
2023-07-2315:16:08 Fine-tune succeeded5.5. Check the fine-tuning progress
You can use two functions provided by openai to check the progress of model fine-tuning.
5.5.1. List all events of fine-tuning tasks
We can use openai.FineTune.list_events() and pass in the fine_tune_response id to list all current events related to the fine-tune task:
fine_tune_events = openai.FineTune.list_events(id=fine_tune_response.id)Output result:
<OpenAIObject list at 0x785e5604fe20> JSON: {
"object": "list",
"data": [
{
"object": "fine-tune-event",
"level": "info",
"message": "Created fine-tune: ft-ByTj1ZNnSn6Fl7rvm8MhzcVV",
"created_at": 1690133686
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune costs $0.06",
"created_at": 1690142386
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune enqueued. Queue number: 1",
"created_at": 1690142386
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune is in the queue. Queue number: 0",
"created_at": 1690142453
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune started",
"created_at": 1690142457
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 1/4",
"created_at": 1690142523
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 2/4",
"created_at": 1690142527
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 3/4",
"created_at": 1690142531
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 4/4",
"created_at": 1690142535
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Uploaded model: curie:ft-personal-2023-07-23-20-02-38",
"created_at": 1690142558
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Uploaded result file: file-CjPz8kFeL0r3Al140byoI02c",
"created_at": 1690142559
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune succeeded",
"created_at": 1690142559
}
]
}5.5.2. Retrieve fine-tuning tasks based on conditions
You can also use openai.FineTune.retrieve(id=fine_tune_response.id) to retrieve the specified fine-tune task, which contains detailed information about the fine-tune task:
retrieve_response = openai.FineTune.retrieve(id=fine_tune_response.id)Output result:
<FineTune fine-tune id=ft-ByTj1ZNnSn6Fl7rvm8MhzcVV at 0x785e2c3fddf0> JSON: {
"object": "fine-tune",
"id": "ft-ByTj1ZNnSn6Fl7rvm8MhzcVV",
"hyperparams": {
"n_epochs": 4,
"batch_size": 1,
"prompt_loss_weight": 0.01,
"learning_rate_multiplier": 0.1
},
"organization_id": "org-MplDvRDOWGVmqFOhYs8cLCMd",
"model": "curie",
"training_files": [
{
"object": "file",
"id": "file-hKjjDPr885IqmsfcDJvEsidl",
"purpose": "fine-tune",
"filename": "file",
"bytes": 15103,
"created_at": 1690133676,
"status": "processed",
"status_details": null
}
],
"validation_files": [],
"result_files": [
{
"object": "file",
"id": "file-CjPz8kFeL0r3Al140byoI02c",
"purpose": "fine-tune-results",
"filename": "compiled_results.csv",
"bytes": 4754,
"created_at": 1690142559,
"status": "processed",
"status_details": null
}
],
"created_at": 1690133686,
"updated_at": 1690142559,
"status": "succeeded",
"fine_tuned_model": "curie:ft-personal-2023-07-23-20-02-38",
"events": [
{
"object": "fine-tune-event",
"level": "info",
"message": "Created fine-tune: ft-ByTj1ZNnSn6Fl7rvm8MhzcVV",
"created_at": 1690133686
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune costs $0.06",
"created_at": 1690142386
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune enqueued. Queue number: 1",
"created_at": 1690142386
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune is in the queue. Queue number: 0",
"created_at": 1690142453
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune started",
"created_at": 1690142457
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 1/4",
"created_at": 1690142523
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 2/4",
"created_at": 1690142527
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 3/4",
"created_at": 1690142531
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Completed epoch 4/4",
"created_at": 1690142535
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Uploaded model: curie:ft-personal-2023-07-23-20-02-38",
"created_at": 1690142558
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Uploaded result file: file-CjPz8kFeL0r3Al140byoI02c",
"created_at": 1690142559
},
{
"object": "fine-tune-event",
"level": "info",
"message": "Fine-tune succeeded",
"created_at": 1690142559
}
]
}5.5.3. Query all fine-tuning task lists
By calling the list() method of OpenAI's FineTune API, you can get a list of models that have been fine-tuned for further analysis and processing.
openai.FineTune.list()5.6. Save the fine-tuned model
Troubleshooting fine_tuned_model is null During fine-tuning, the fine_tuned_model may not be immediately available in the fine_tune_response object returned by openai.FineTune.create().
To check the status of the fine-tuning process, we can call the openai.FineTune.retrieve() function and pass in the fine_tune_response.id . The function will return a JSON object containing information about the state of the training, such as the current epoch, current batch, training loss, and validation loss.
After the fine-tuning process is complete, you can check the status of all fine-tuned models by calling openai.FineTune.list(). This will list all your tweaks and their current status.
After the fine-tuning process is complete, you can retrieve the fine_tuned_model key by calling opena.FineTune.retrieve(). Run it again and pass in fine_tune_response.id . This will return a JSON object containing the key fine_tuned_model and the ID of the fine-tuned model that can be used for further completion.
After fine-tuning is complete, go ahead and save the name of the fine-tuned model:
# 如果response.fine_tuned_model不为空
fine_tuned_model = fine_tune_response.fine_tuned_model
fine_tuned_model
# 如果response.fine_tuned_model为空
retrieve_response = openai.FineTune.retrieve(fine_tune_response.id)
fine_tuned_model = retrieve_response.fine_tuned_model
fine_tuned_modelSo far, we have completed a new model after fine-tuning, and then use the new model to test it.
5.7. Testing new models
When the fine-tuning task is executed successfully, the fine_tuned_model field will be filled with the name of the model. We can then specify this model as a parameter to our Completions API and make a request.
After the fine-tuning task completes for the first time, it might take several minutes for the trained model to be ready to handle requests. If a request to a model times out, it may be because the model is loading. If this happens, please try again in a few minutes.
Using this model, we can verify its results by providing a hint, a model name, and running the query to create it using the function openai.Completion.create(). The results retrieved from the answer dictionary are as follows:
new_prompt = "新中国的创始人是谁?"
answer = openai.Completion.create(
model=fine_tuned_model,
prompt=new_prompt
)
print(answer['choices'][0]['text'])
new_prompt = """古代丝绸之路的起点在深圳哪个城市?"""
answer = openai.Completion.create(
model=fine_tuned_model,
prompt=new_prompt
)
print(answer['choices'][0]['text'])Even though the prompts were not written exactly as in the validation dataset, the model still managed to map them to the correct answer. With very few training samples, we managed to build a decent fine-tuned model. Larger training sizes lead to better results.
For these fine-tuned model requests, you can continue to use all other Completions parameters, such as temperature, frequency_penalty, etc. presence_penalty.
5.8. Delete fine-tuning model
openai.Model.delete(FINE_TUNED_MODEL)6. The cost of model fine-tuning
The base model we chose here is Curie, not Davinci, which works best. The reason for making such a choice is due to cost considerations. The figure below shows the detailed information about fine-tuning rates for each model.
The cost of using a fine-tuned model is much higher than using OpenAI's built-in models. The fine-tuned model based on Davinci, when used, costs $0.12 per 1000 Tokens, 6 times that of using the built-in text-davinci-003, and 60 times that of our most commonly used gpt-3.5-turbo. Therefore, if it is only a very basic application, the cost is too high. Even if we choose to fine-tune based on Curie, the cost of using 1000 Tokens is $0.012, although it is cheaper than text-davinci-003, but it is also 6 times that of gpt-3.5-turbo.
6.1. Incremental training and optimization model
Although the relatively high price of the fine-tuned model limits its use. However, there is another ability to fine-tune the model, that is, we can further fine-tune the fine-tuned model based on new data. This is very useful in many vertical fields. For example, in fields such as medicine and finance, we can continuously collect new data and continue to fine-tune our model on the basis of the previous fine-tuning model, so that the effect of the model is getting better and better. good. And these areas can often bear higher costs. Further fine-tuning is actually not complicated to operate, just prepare some data, and use the previously fine-tuned model as the basic model to operate.
When fine-tuning on the original model, we need to modify two parameters.
- The first is the model parameter. We replaced Curie with the model curie:ft-personal-2023-07-23-20-02-38 after fine-tuning.
- The second is learning_rate_multiplier, the default value of this parameter is from 0.05 to 0.2 according to your sample size. If the number of samples you continue to fine-tune is much smaller than the amount of data you fine-tuned before, you can make it larger.
After the fine-tuning is updated, the name of the model remains unchanged, and the old model is updated to the new fine-tuned model.
7. Conclusion
Today we mainly introduced how to use OpenAI's API to fine-tune the GPT model. Model fine-tuning provides us with a very practical ability. We can use our own data to adjust model parameters to generate a new model based on the OpenAI basic model. In this way, we can produce a model exclusive to ourselves based on the data of our own proprietary vertical field. Moreover, we can continue to fine-tune iterations on this model based on newly collected data. However, the cost of using the fine-tuned model is relatively high. You need to calculate for yourself whether the ROI of the fine-tuned model is relatively high, or it is more cost-effective to use the previous external knowledge base.
8. Recommended reading
1. OpenAI's model fine-tuning actually has many richer usages, such as classification or named entity extraction. You can go to the Specific Guidelines section of the official website to take a look and try to fine-tune a model.
- https://platform.openai.com/docs/guides/fine-tuning
2. In its official documents, OpenAI recommends the company Weight & Bias, which can track the experiments, models and data sets of the fine-tuned model. Weight & Bias also provides a Notebook for model fine-tuning on the WIT dataset in its own document. If you are interested, you can also take a look.
- https://platform.openai.com/docs/guides/fine-tuning/weights-biases
- https://docs.wandb.ai/guides/integrations/openai?utm_source=wandb_docs&utm_medium=code&utm_campaign=OpenAI+API
3. Some users who use Microsoft Azure cloud services can refer to Microsoft's official documents for how to fine-tune their models based on Azure OpenAI
- https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/fine-tuning?pivots=programming-language-studio
If you are interested in this article, and you want to learn more about practical skills in the field of AI , you can pay attention to the "Technology Craze AI" public account . Here, you can see the latest and hottest dry goods articles and case tutorials in the AIGC field.