First of all, let me explain our MOSS version. The current open source version is called MOSS 003. In February, the open-source version is called MOSS 002. In January, we also have an internal test version called OpenChat 001. Here is a brief introduction to our MOSS version. successive iterations.
OpenChat 001
After ChatGPT came out last year, domestic NLP practitioners were greatly impacted. At that time, there was no llama or alpaca. It is generally believed that we are one to two years away from ChatGPT. It is very expensive to do two parts of ChatGPT, one is data labeling, and the other is pre-training computing power. We don’t have computing power, but we can try to construct some data. After all, AI is powerful enough to replace the work of so many people. There is no reason to think that it cannot replace the work of labelers. So we picked up some user prompts collected by their API from the appendix of OpenAI's paper, and then used text-davinci-003 to expand about 400,000 dialogue data with an idea similar to Self-Instruct. That's right, it's similar to today's alpaca, and we were still having multiple rounds of dialogue instead of single rounds of instructions. After that, I did a fine-tune on the 16B base (CodeGen), and found that it seems that the slightly larger model can easily learn the ability to follow instructions. The following are some examples at that time.
OpenChat 001 Directive Follow Ability
OpenChat 001 Multi-round dialogue capability