Meet Stable Beluga 1 and Stable Beluga 2, Our Large and ...

Meet Stable Beluga 1 and Stable Beluga 2, Our Large and Mighty Instruction Fine-Tuned Language Models — Stability AI Meet Stable Beluga 1 and Stable Beluga 2, Our Large and Mighty Instruction Fine-Tuned Language Models Jul 21 Updated 28 Jul 2023 Stability AI and its CarperAI lab proudly announce Stable Beluga 1 and its successor Stable Beluga 2(formerly codenamed FreeWilly), two powerful new, open access, Large Language Models (LLMs). Both models demonstrate exceptional reasoning ability across varied benchmarks. Stable Beluga 1 leverages the original LLaMA 65B foundation model and was carefully fine-tuned with a new synthetically-generated dataset using Supervised Fine-Tune (SFT) in standard Alpaca format. Similarly, Stable Beluga 2 leverages the LLaMA 2 70B foundation model to achieve industry-leading performance. Both models are research experiments and are released to foster open research under a non-commercial license. While we have conducted internal red-teaming to ensure the model remains polite and harmless, we welcome the community's feedback and help in further red-teaming. Data Generation and Collection The training for the Stable Beluga models was directly inspired by the methodology pioneered by Microsoft in its paper: "Orca: Progressive Learning from Complex Explanation Traces of GPT-4.” While our data generation process is similar, we differ in our data sources. Our variant of the dataset, containing 600,000 data points (roughly 10% of the dataset size the original Orca paper used), was created synthetically using high-quality instructions from the following datasets created by Enrico Shippole: FLAN 2021 Submix Original To ensure fair comparisons, we carefully filtered these datasets and removed examples that originated from evaluation benchmarks. Despite training on one-tenth the sample size of the original Orca paper (significantly reducing the cost and carbon footprint of training the model compared to the original paper), the r