StarCoder2 and The Stack v2StarCoder2BigCode is releasing... | StarCoder2 and The Stack v2StarCoder2BigCode is releasing...
StarCoder2 and The Stack v2
StarCoder2
BigCode is releasing StarCoder2, the next generation of transparently trained open code LLMs. All StarCoder2 variants were trained on The Stack v2, a new large and high-quality code dataset. We release all models, datasets, and the processing as well as the training code. Check out the paper for details.

What is StarCoder2?
StarCoder2 is a family of open LLMs for code and comes in 3 different sizes with 3B, 7B and 15B parameters. The flagship StarCoder2-15B model is trained on over 4 trillion tokens and 600+ programming languages from The Stack v2. All models use Grouped Query Attention, a context window of 16,384 tokens with a sliding window attention of 4,096 tokens, and were trained using the Fill-in-the-Middle objective. https://github.com/huggingface/blog/blob/main/starcoder2.md #StarCoder2 blog/starcoder2.md at main · huggingface/blog