TinyLlama, the mini AI model trained on a trillion tokens

 Compact, but very powerful. Petit Lama , the highly anticipated open source model, is here. Weighing less than 640 megabytes, the model was trained on a trillion tokens and outperforms competitors of similar size.

The TinyLlama project began last September, when a group of developers began training a small model on billions of tokens. After a lot of work and some setbacks, the TinyLlama team has now released the model. THE model has a size of 1 billion parameters and was trained on around a trillion tokens for around three cycles via the training data.

TinyLlama outperforms other “open source” “ comparable  ” LLMs 

According to the paper describing the model, the finished TinyLlama outperforms existing open source language models of comparable size, including Pythia-1.4B, OPT-1.3B, and MPT-1.3B.

TinyLlama's potential use cases were able to see the model deployed to edge devices, as it only takes up 637MB . The model could also be used to aid speculative decoding of larger patterns; The team that created it referenced a tutorial from former Tesla senior director of artificial intelligence Andrei Karpathy who now works at OpenAI.

The model itself was designed to be a compact version of Lama 2 , Half 's open source language model , which even has the same architecture and tokenizer, meaning it can be plugged in and used in projects based on on Llama.

Despite its small size, TinyLlama can be used for downstream tasks, and the team that created it calls it "an attractive platform for researchers and practitioners of language modeling research."

For example, Awni Hannoun , an Apple machine learning researcher, built TinyLlama with LoRA locally, using just an 8GB Mac Mini, via MLX , Apple's suite of open source training tools.

“With its compact architecture and promising performance, TinyLlama can enable end-user applications on mobile devices and serve as a lightweight platform for testing a wide range of innovative language model ideas,” said l team behind the mini-model.

And more TinyLlamas are on the way: the developers plan to develop "enhanced versions", including increased performance and versatility in various tasks.

How to access TinyLlama

You can download TinyLlama for free via GitHub. All checkpoint models are also available. TinyLlama can be used for commercial purposes under the Apache-2.0 license.

The team that created the model recommends using the chat version of TinyLlama, because the learning rate of the base model "hasn't cooled down yet."

Little models grow

A wave of smaller AI models has emerged recently as companies seek to reduce the cost of running hardware.

Microsoft, for example, has its own Phi project that works on smaller models, a few billion parameters wide, but capable of beating the bigger ones. Phi-2, launched last December, outperformed models up to 25 times larger.

The launch of Gemini Nano, the smaller version of Google 's new flagship model which will have a size of around 3.2 billion parameters, is expected later this year.

According to Bradley Shimmin, principal analyst for AI and data analytics at research firm Omdia, these small models work well because they are trained on synthetic data generated by larger models.

“Synthetic data is already driving a lot of the innovation we're seeing coming from the generative AI space itself, where there are many smaller models currently impressing people with their capabilities that match those pioneering models like OpenAI's GPT