Just now, Meta AI released Code Llama as expected, a large language model (LLM) that can generate code using text prompts. Code Llama is a state-of-the-art open LLM for coding tasks that has the potential to make current developer workflows faster and more efficient and lower the barrier to entry for those learning to code. Code Llama has the potential to be used as a productivity and educational tool to help programmers write more robust and well-documented software.
The exciting thing is that Code Llama is completely open source and is now open for research and commercial use.
1. Meta AI News
Meta AI's recent blockbuster actions have been frequent, and a series of open source large models have been released in just over a month. Let's take a look at the influential products.
On July 14, 2023, Meta AI is proud to introduce CM3leon, the first multimodal model that achieves state-of-the-art text-to-image generation performance and is 5x more computationally efficient than competing models.
On July 18, 2023, Meta and Microsoft launched the next generation of Llama, Llama 2 is free for research and commercial use.
Llama 2 is Meta's open source Large Language Model (LLM). It's basically Facebook's parent company's answer to OpenAI's GPT model and Google's AI models like PaLM 2, but with one key difference: it's almost free for anyone to use for research and commercial purposes. August 16, 2023
On August 2, 2023, Meta's parent company, Facebook, unveiled a new generative artificial intelligence tool called AudioCraft, which allows users to create high-quality audio and music using text prompts. The tool includes audio models MusicGen, AudioGen, and EnCodec, which can generate music and audio from text prompts.
AudioCraft consists of three models: MusicGen, AudioGen, and EnCodec. MusicGen was trained on Meta-owned and exclusively licensed music to generate music from text prompts, while AudioGen was trained on public sound effects to generate audio from text prompts.
On August 23, 2023, Meta AI proudly presents SeamlessM4T, the first all-in-one multilingual multimodal translation model. This single model can perform speech-to-text, speech-to-speech, text-to-text translation and speech recognition tasks in up to 100 languages depending on the task.
On the same day, MetaAI's new SeamlessM4T model is now available on Hugging Face!
On August 24, 2023, according to The Information, Meta released the open source code generation AI model Code Llama on Thursday (August 24). The model is designed to help developers automatically recommend code snippets when writing code to improve development efficiency, and it also aims to make it easier for companies to create AI assistants.
2. How Code Llama works
Code Llama is a code-specific version of Llama 2, created by further training Llama 2 on its code-specific dataset, sampling more data from the same dataset for longer periods of time. Essentially, Code Llama has enhanced coding capabilities, built on top of Llama 2. It can generate code and natural language about the code based on code and natural language cues (eg, "write me a function that outputs the Fibonacci sequence.") It can also be used for code completion and debugging. It supports many of the most popular languages in use today, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash (see our research paper for a full list).
Meta released the Code Llama in three sizes, with 7B, 13B and 34B parameters. Each model is trained with 500B code tokens and code related data. The 7B and 13B base and instruction models are also trained with Fill-in-the-Intermediate (FIM) capabilities, allowing them to insert code into existing code, which means they can support tasks like code completion out of the box.
These three models meet different service and latency requirements. For example, the 7B model can run on a single GPU. The 34B model returns the best results and provides better coding assistance, but the smaller 7B and 13B models are faster and better suited for tasks that require low latency, such as real-time code completion.
The Code Llama model provides robust generation with up to 100,000 context tokens. All models are trained on 16,000 labeled sequences and show improvement on inputs up to 100,000 labeled.
In addition to being a prerequisite for generating longer programs, having longer input sequences can unlock exciting new use cases for Code LLM. For example, users can provide models with more context from their codebase to make generations more relevant. It also helps to debug scenarios in larger code bases, where it can be challenging for developers to have all the code relevant to a specific problem at hand. When developers are faced with debugging large amounts of code, they can pass the entire code length into the model.
In addition, Meta has further fine-tuned two additional variants of Code Llama: Code Llama - Python and Code Llama - Instruct.
Code Llama - Python is a language-specific variant of Code Llama, further fine-tuned on the 100B mark of Python code. Because Python is the most benchmarked language for code generation, and because Python and PyTorch play important roles in the AI community, we believe specialized models can provide additional utility.
Code Llama - Instruct is an instruction trimming and alignment variant of Code Llama. Instruction adjustments continue the training process, but with different goals. The model accepts "natural language instructions" as input and expected output. This allows it to better understand what people expect from the prompt. We recommend using the Code Llama - Instruct variant when using Code Llama for code generation, because Code Llama - Instruct has been fine-tuned to generate useful and safe answers in natural language.
Meta does not recommend using Code Llama or Code Llama - Python for general natural language tasks because neither model is designed to follow natural language instructions. Code Llama is designed for code-specific tasks and is not suitable as a base model for other tasks.
When using Code Llama models, users must abide by our License and Acceptable Use Policy.
3. Evaluate the performance of Code Llama
To test Code Llama's performance against existing solutions, we used two popular coding benchmarks: HumanEval and Mostly Basic Python Programming (MBPP). HumanEval tests the model's ability to complete code from docstrings, and MBPP tests the model's ability to write code from descriptions.
Our benchmarks show that Code Llama outperforms open-source, code-specific Llama, and outperforms Llama 2. For example, Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP, the highest compared to other states. State-of-the-art open solution, comparable to ChatGPT.
As with all cutting-edge technology, Code Llama comes with risks. Building AI models responsibly is critical, and we took a number of safety measures before releasing Code Llama. As part of our red team efforts, we performed a quantitative assessment of the risk of Code Llama generating malicious code. We created hints in an attempt to solicit malicious code with a clear intent and scored Code Llama's responses to these hints according to ChatGPT (GPT3.5 Turbo). Our results found that Code Llama's answer was more secure.
See our research papers for details on responsible red team work by experts in the fields of artificial intelligence, offensive security engineering, malware development, and software engineering.
4. References
Code Llama GitHub：https://github.com/facebookresearch/codellama
Code Llama Model：https://ai.meta.com/resources/models-and-libraries/llama-downloads/
Code Llama Pager：https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/
来源：Meta AI：https://ai.meta.com/blog/code-llama-large-language-model-coding/