🐣Ai2 Releasing OLMoE!
OLMoE-1B-7B-Instruct is a Mixture-of-Experts LLM with 1B active and 7B total parameters, and, OLMoE is 100% open-source in model, code-base, datasets!

🦖Paper: https://arxiv.org/abs/2409.02060

🤗Model:
allenai/OLMoE-1B-7B-0924-Instruct


💾Datasets:
allenai/OLMoE-mix-0924


🙇‍♂️Demo:
vilarin/OLMoE https://huggingface.co/spaces/vilarin/OLMoE

arXiv.org

OLMoE: Open Mixture-of-Experts Language Models

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain...