Share and discover more about AI with social posts from the community.huggingface/OpenAi
# Excited to Share: New LLM Tokenization - Convert Text to tokens and vice versa! 🚀
I've just developed a powerful tool for anyone working with Language Models (LLMs) or diving into Natural Language Processing (NLP).
🔍 Introducing the LLM Tokenization - Convert Text to tokens and vice versa!!
Key Features:
- Convert text to tokens and token IDs
- Reverse engineer: convert token IDs back to text
- Support for popular models: LLama3 (Will add more models iteratively)
- User-friendly Gradio interface for easy interaction
Whether you're debugging your NLP pipeline, exploring how different models tokenize text, or just curious about the inner workings of LLMs, this tool is for you!
👩💻 Tech Stack:
- Python
- Gradio for the web interface
- Hugging Face Transformers for tokenization
The application is deployed in Hugging Face spaces as Gradio application
🔗 Try it out: https://lnkd.in/g6R5z9k2
#NLP #MachineLearning #AI #PythonDevelopment #OpenSourceAI
I've just developed a powerful tool for anyone working with Language Models (LLMs) or diving into Natural Language Processing (NLP).
🔍 Introducing the LLM Tokenization - Convert Text to tokens and vice versa!!
Key Features:
- Convert text to tokens and token IDs
- Reverse engineer: convert token IDs back to text
- Support for popular models: LLama3 (Will add more models iteratively)
- User-friendly Gradio interface for easy interaction
Whether you're debugging your NLP pipeline, exploring how different models tokenize text, or just curious about the inner workings of LLMs, this tool is for you!
👩💻 Tech Stack:
- Python
- Gradio for the web interface
- Hugging Face Transformers for tokenization
The application is deployed in Hugging Face spaces as Gradio application
🔗 Try it out: https://lnkd.in/g6R5z9k2
#NLP #MachineLearning #AI #PythonDevelopment #OpenSourceAI
I just had a masterclass in open-source collaboration with the release of Llama 3.1 🦙🤗
Meta dropped Llama 3.1, and seeing firsthand the Hugging Face team working to integrate it is nothing short of impressive. Their swift integration, comprehensive documentation, and innovative tools showcase the power of open-source teamwork.
For the curious minds:
📊 Check out independent evaluations:
open-llm-leaderboard/open_llm_leaderboard
🧠 Deep dive into the tech: https://huggingface.co/blog/llama31
👨🍳 Try different recipes (including running 8B on free Colab!): https://github.com/huggingface/huggingface-llama-recipes
📈 Visualize open vs. closed LLM progress:
andrewrreed/closed-vs-open-arena-elo
🤖 Generate synthetic data with distilabel, thanks to the new license allowing the use of outputs to train other LLMs https://huggingface.co/blog/llama31#synthetic-data-generation-with-distilabel
💡 Pro tip: Experience the 405B version for free on HuggingChat, now with tool-calling capabilities! https://huggingface.co/chat/
#OpenSourceAI #AIInnovation
Meta dropped Llama 3.1, and seeing firsthand the Hugging Face team working to integrate it is nothing short of impressive. Their swift integration, comprehensive documentation, and innovative tools showcase the power of open-source teamwork.
For the curious minds:
📊 Check out independent evaluations:
open-llm-leaderboard/open_llm_leaderboard
🧠 Deep dive into the tech: https://huggingface.co/blog/llama31
👨🍳 Try different recipes (including running 8B on free Colab!): https://github.com/huggingface/huggingface-llama-recipes
📈 Visualize open vs. closed LLM progress:
andrewrreed/closed-vs-open-arena-elo
🤖 Generate synthetic data with distilabel, thanks to the new license allowing the use of outputs to train other LLMs https://huggingface.co/blog/llama31#synthetic-data-generation-with-distilabel
💡 Pro tip: Experience the 405B version for free on HuggingChat, now with tool-calling capabilities! https://huggingface.co/chat/
#OpenSourceAI #AIInnovation
huggingface.co
Llama 3.1 - 405B, 70B & 8B with multilinguality and long context
We’re on a journey to advance and democratize artificial intelligence through open source and open science.