LLMs scaling instruction model Scaling instruction models FLAN (Fine-tuned LAnguage Net, fine-tuned language network)

This paper introduces FLAN (Fine-tuned LAnguage Net), a guided fine-tuning method, and presents the results of its application. The study demonstrates that by fine-tuning the 540B PaLM model on 1836 tasks while integrating Chain-of-Thought Reasoning data, FLAN achieves improvements over the base model in terms of generalization, human usability, and zero-shot reasoning. The paper also details how to assess these aspects.

Here are images from the lecture slides illustrating the fine-tuning task and dataset used when training FLAN. Task selection expands on previous work by incorporating dialogue and procedural synthesis tasks from Muffin and integrating them with the new Chain of Thought Reasoning task. It also includes subsets of other task collections such as T0 and Natural Instructions v2. Some tasks are retained during training and later used to evaluate the model's performance on unseen tasks.

reference

https://www.coursera.org/learn/generative-ai-with-llms/supplement/aDQwy/scaling-instruct-models