foreword
ChatGPT is a classic performance of "making miracles with great effort". The large model brings amazing intelligence to ChatGPT, but it is very expensive to train such a large model. According to the data given by OpenAI, the Davinci model with 170 billion parameters was rebuilt from scratch. It will take about 3 months and cost 1.5 million US dollars to train once. Then we ordinary people or small companies face this high threshold, is there no hope for custom models? In fact, in addition to training a model from scratch, we can also choose to train based on a basic model. In this way, we can add our own personalized data to it, and finally get a personalized model with domain enhancement. This technology is called Fine-tuning by OpenAI. .
What is the use of the personalization model? We know that the models given by OpenAI (such as Davinci, Curie, gpt-3.5-turbo) are general models, and the industries and knowledge in modern society are so huge that each field has its own subdivided expertise, for example, We know that a typical application scenario of ChatGPT is intelligent customer service, but it is also customer service. Customer service in the insurance field and customer service in Taobao stores have completely different customers and questions to be answered. If you want to give better answers, We need to polish our own personalized model.
principle
OpenAI's Fine-tuning technology is based on a pre-trained model, which can quickly adapt to specific tasks by fine-tuning the model on small-scale tasks. Its core principle is to fine-tune the weights of the pre-trained model to adapt to the input and output of new tasks.
Fine-tuning generally includes the following steps:
- Load a pre-trained model: Select a pre-trained model related to the desired task and load its weights.
- Select Task Dataset: Select the required dataset for a specific task.
- Fine-tune the model: take the task dataset as input to minimize the loss function of the model on this dataset. In this process, it is usually necessary to perform multiple iterations on the training set and validation set to avoid overfitting problems.
- Test on the test set: Use the fine-tuned model to test its performance on the test set.
The advantage of fine-tuning is that it allows us to quickly obtain a high-quality model for a specific task without training the model from scratch. At the same time, since Fine-tuning only updates part of the weights, and most of the weights have been well optimized in the pre-training stage, compared with training the model from scratch, Fine-tuning requires more training time and computing resources. much less.
step
Here, I thought of an idea. Isn’t the question and answer of the knowledge planet’s fan group the best example? It may be more interesting to train the data and see the results.
a. Prepare training data
We need to prepare the data to be trained in jsonl format. The characteristic of this format is that each row is in json format (about what is json, refer to here ) as follows
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}For example, my training data is dataset.jsonlabout 2,500 lines of questions and answers. For fine-tuning, the amount of data should not be too small. If there are less than 100 pieces of data, the quality of the trained model should not be good:
{"prompt": "医药和消费有什么基金推荐的", "completion": "医药可以看看工银前沿的赵蓓，她挺均衡的，对于这个行业我了解不多，你还可以看看医药100指数，消费挺多的，消费50也挺好。"}
{"prompt": "请教一下老师，恒生科技第一大持仓股是快手。而快手是亏损最大的互联网企业。似乎齐老师也说过不看好快手，会不会影响恒生科技持仓。", "completion": "如果你要是能选股，确实不应该买指数。从指数选择的角度来说。中概互联我们更看好一些。但他跟恒生科技的相关度其实很高"}
{"prompt": "想问一下国投瑞银这边基金公司和綦缚鹏的风格实力怎么样", "completion": "他风格不是很固定。最近在偏向周期。（个人观点，不作为投资建议）"}Among them, prompt is the question and completion is the answer.
b. Install tools
We all know that in the field of machine learning, the Python language is the main force (refer to here ), and OpenAI is no exception. We need to install the Python language environment first, and the installation step is omitted. After the installation is complete, you need to use the tools that come with python pipto install the training tools provided by OpenAI
pip install --upgrade openaiAt the same time, because we need to interact with the OpenAI platform, we need OPENAI_API_KEY, which requires registration and charges
export OPENAI_API_KEY="<OPENAI_API_KEY>"c. Tuning data
openai tools fine_tunes.prepare_data -f dataset.jsonlThis instruction will help us optimize the training data. During the operation of this instruction, we will ask us a few questions, mainly adding a fixed suffix to prompt, such as "->", adding a space at the beginning and a newline at the end to completion, as follows
{"prompt":"医药和消费有什么基金推荐的 ->","completion":" 医药可以看看工银前沿的赵蓓，她挺均衡的，对于这个行业我了解不多，你还可以看看医药100指数，消费挺多的，消费50也挺好。\n"}
{"prompt":"请教一下老师，恒生科技第一大持仓股是快手。而快手是亏损最大的互联网企业。似乎齐老师也说过不看好快手，会不会影响恒生科技持仓。 ->","completion":" 如果你要是能选股，确实不应该买指数。从指数选择的角度来说。中概互联我们更看好一些。但他跟恒生科技的相关度其实很高\n"}
{"prompt":"想问一下国投瑞银这边基金公司和綦缚鹏的风格实力怎么样 ->","completion":" 他风格不是很固定。最近在偏向周期。（个人观点，不作为投资建议）\n"}The prepared data file is dataset_prepared.jsonl.
d. Submit data for training
After we prepare the data, we will submit it to the OpenAI platform for training. At this time, we need to provide the data file dataset_prepared.jsonl and the model. Currently, there are 4 BASE_MODELs supported by OpenAI (refer to here), which are Davinci, Curie, Ada , Babbage, from left to right, the prices are from expensive to cheap, and the performance is from good to average. If you want to know the effect of each model, you can train based on 2 BASE_MODEL at the same time, and finally compare the effect. Considering the training cost this time, first use Curie as BASE_MODEL training.
openai api fine_tunes.create -t dataset_prepared.jsonl -m curieIt takes about 30 minutes or more. The training progress can be obtained through the following command, ft-SSIJ4DsHFfp9LEtuHWyQcn5B This is the job ID of fine-tuning, which is given by the above create command.
openai api fine_tunes.follow -i ft-SSIJ4DsHFfp9LEtuHWyQcn5BHere is a little trick. If the training data set is large, you don’t have to re-upload the data every time you submit a training task. You can directly use the data that has been uploaded before. The method of use is as follows
openai api fine_tunes.create -t file-WSkHmtSfBLORMAhEoEVyBDO4 -m curieThis file-WSkHmtSfBLORMAhEoEVyBDO4is obtained by the following command after uploading the file before
(.venv) ➜ openai api files.list
{
"data": [
{
"bytes": 620175,
"created_at": 1680619086,
"filename": "dataset_prepared.jsonl",
"id": "file-WSkHmtSfBLORMAhEoEVyBDO4",
"object": "file",
"purpose": "fine-tune",
"status": "processed",
"status_details": null
},
],
"object": "list"
}e. Use the model
View the details of the fine-tuning task
(.venv) ➜ openai api fine_tunes.list
{
"data": [
{
"created_at": 1680619087,
"fine_tuned_model": "curie:ft-personal-2023-04-04-15-28-34",
"hyperparams": {
"batch_size": 2,
"learning_rate_multiplier": 0.1,
"n_epochs": 4,
"prompt_loss_weight": 0.01
},
"id": "ft-SSIJ4DsHFfp9LEtuHWyQcn5B",
"model": "curie",
"object": "fine-tune",
"organization_id": "org-WTfVkVyUtqWdmv711ZZWHw7T",
"result_files": [
{
"bytes": 225243,
"created_at": 1680622115,
"filename": "compiled_results.csv",
"id": "file-RfmMXqxf94DhsxLN2jfATCTW",
"object": "file",
"purpose": "fine-tune-results",
"status": "processed",
"status_details": null
}
],
"status": "succeeded",
"training_files": [
{
"bytes": 620175,
"created_at": 1680619086,
"filename": "dataset_prepared.jsonl",
"id": "file-WSkHmtSfBLORMAhEoEVyBDO4",
"object": "file",
"purpose": "fine-tune",
"status": "processed",
"status_details": null
}
],
"updated_at": 1680622115,
"validation_files": []
},
],
"object": "list"
}
(.venv) ➜ openai api fine_tunes.follow -i ft-SSIJ4DsHFfp9LEtuHWyQcn5B
[2023-04-04 22:38:07] Created fine-tune: ft-SSIJ4DsHFfp9LEtuHWyQcn5B
[2023-04-04 22:46:52] Fine-tune costs $4.89
[2023-04-04 22:46:53] Fine-tune enqueued. Queue number: 5
[2023-04-04 22:47:38] Fine-tune is in the queue. Queue number: 4
[2023-04-04 22:50:08] Fine-tune is in the queue. Queue number: 3
[2023-04-04 22:50:15] Fine-tune is in the queue. Queue number: 2
[2023-04-04 22:54:51] Fine-tune is in the queue. Queue number: 1
[2023-04-04 22:57:26] Fine-tune is in the queue. Queue number: 0
[2023-04-04 22:57:50] Fine-tune started
[2023-04-04 23:06:11] Completed epoch 1/4
[2023-04-04 23:20:51] Completed epoch 3/4
[2023-04-04 23:28:34] Uploaded model: curie:ft-personal-2023-04-04-15-28-34
[2023-04-04 23:28:35] Uploaded result file: file-RfmMXqxf94DhsxLN2jfATCTW
[2023-04-04 23:28:35] Fine-tune succeeded
Job complete! Status: succeeded
Try out your fine-tuned model:
openai api completions.create -m curie:ft-personal-2023-04-04-15-28-34 -p <YOUR_PROMPT> After the model training is complete, the custom model obtained is "curie:ft-personal-2023-04-04-15-28-34". It took 20 minutes to queue up and 30 minutes to train, costing about $4.89, 2500 pieces Data is not cheap.
Next, we use the new model to ask questions
(.venv) ➜ openai api completions.create -m "curie:ft-personal-2023-04-04-15-28-34" -p "医药和消费有什么基金推荐的 ->"
》医药和消费有什么基金推荐的 -> 消费和医药比% Unfortunately, the answer given is relatively poor. We look at the training result data. The ultimate goal of our training is to minimize the result of the loss function, that is, the training_loss value is the smallest, and the training_sequence_accuracy and training_token_accuracy values are the largest, but from the following data, training_loss The minimum is 0.06372293254171614, the training_sequence_accuracy is 0.5, and the highest training_token_accuracy is only 0.9166666666666666, and the performance is very unstable, which also confirms why the effect of question and answer is not good.
(.venv) ➜ openai api fine_tunes.results -i "ft-SSIJ4DsHFfp9LEtuHWyQcn5B"
step,elapsed_tokens,elapsed_examples,training_loss,training_sequence_accuracy,training_token_accuracy
1,482,2,0.45391547040630686,0.0,0.5755395683453237
2,980,4,0.6011948533307662,0.0,0.6086956521739131
3,1334,6,0.5914613516018767,0.0,0.528
4,1640,8,0.41827734171512065,0.0,0.5526315789473685
5,2618,10,0.5738252286081992,0.0,0.6934673366834171
6,3148,12,0.32921856693228024,0.0,0.639344262295082
7,3550,14,0.5036230062589969,0.0,0.5983606557377049
3855,2231118,7710,0.06739243882282422,0.0,0.9139784946236559
......
......
省略了中间的几千行内容
......
......
3856,2231616,7712,0.16957119258711775,0.0,0.8467153284671532
3857,2232082,7714,0.2004900809357239,0.5,0.808
3858,2232276,7716,0.08515305384741474,0.0,0.8518518518518519
3859,2232678,7718,0.19010555123939107,0.0,0.8529411764705882
3860,2233608,7720,0.057206735256356465,0.0,0.8762886597938144
3861,2234074,7722,0.2355557036819172,0.0,0.7916666666666666
3862,2234396,7724,0.14572308102632223,0.0,0.905511811023622
3863,2235182,7726,0.13363321007470183,0.0,0.8378378378378378
3864,2235488,7728,0.09592527983439035,0.5,0.9166666666666666
3865,2236002,7730,0.06372293254171614,0.0,0.8909090909090909Well, the next step is to locate specific problems and then perform tuning, which will be discussed in the next article.
in conclusion
In this article, the basic concepts and principles of model fine-tuning are described, as well as a quick way to use OpenAI's tools to train your own model. Overall, the quick start is relatively simple, but it still takes a lot of time and money to adjust the model. The curie demonstrated above is not the most expensive basic model. The most expensive is the davinci model, which is even more expensive. 10 times.