UAE’s Technology Innovation Institute Revolutionises AI Language Models with New Architecture

The Falcon Mamba 7B is the world’s top-performing open-source State Space Language Model (SSLM), as independently verified by Hugging Face. SSLMs have a low memory cost and do not require additional memory to generate arbitrarily long blocks of text. Falcon Mamba 7B also outperforms traditional transformer architecture models such as Meta’s Llama 3.1 8B and Mistral’s 7B. The new model reflects the innovation and pioneering approach of Abu Dhabi in AI research and development.

The Technology Innovation Institute (TII), a leading global scientific research centre and the applied research pillar of Abu Dhabi’s Advanced Technology Research Council (ATRC), has released a new large language model in its Falcon series, the Falcon Mamba 7B. The new model is the world’s top-performing open-source State Space Language Model (SSLM), as independently verified by Hugging Face.

As the first SSLM for Falcon, it departs from previous Falcon models, which all utilised a transformer-based architecture. This new Falcon Mamba 7B model exemplifies the pioneering research the institute is conducting and the breakthrough tools and products it provides to the community in an open-source format.

H.E. Faisal Al Bannai, Secretary General of ATRC and Adviser to the UAE President for Strategic Research and Advanced Technology Affairs, said: “The Falcon Mamba 7B marks TII’s fourth consecutive top-ranked AI model, reinforcing Abu Dhabi as a global hub for AI research and development. This achievement highlights the UAE’s unwavering commitment to innovation.”

In terms of transformer architecture models, Falcon Mamba 7B outperforms Meta’s Llama 3.1 8B, Llama 3 8B, and Mistral’s 7B on the newly introduced benchmarks from Hugging Face. Additionally, among other SSLMs, Falcon Mamba 7B surpasses all other open-source models on the old benchmarks and will be the first model on Hugging Face’s new, more challenging benchmark leaderboard.

Dr. Najwa Aaraj, Chief Executive of TII, commented: “The Technology Innovation Institute continues to push the boundaries of technology with its Falcon series of AI models. The Falcon Mamba 7B represents true pioneering work and paves the way for future AI innovations that will enhance human capabilities and improve lives.”

State Space models are highly effective at understanding complex situations that evolve over time, such as an entire book. This is because SSLMs do not require additional memory to process such large volumes of information.

On the other hand, transformer-based models are very efficient at remembering and using information they have previously processed in a sequence, making them highly effective at tasks like content generation. However, because they compare every word with every other word, they require significant computational power.

SSLMs can be applied in various fields such as estimation, forecasting, and control tasks. Similar to transformer architecture models, they also excel in Natural Language Processing tasks and can be used in machine translation software, text summarisation, computer vision, and audio processing..

Dr. Hakim Hacid, Acting Chief Researcher of TII’s AI Cross-Centre Unit, said: “As we introduce the Falcon Mamba 7B, I’m proud of the collaborative ecosystem at TII that nurtured its development. This release represents a significant stride forward, inspiring fresh perspectives and further fuelling the quest for intelligent systems. At TII, we’re pushing the boundaries of both SSLM and transformer models to spark further innovation in generative AI.”

Falcon LLMs have been downloaded over 45 million times, demonstrating the outstanding success of the models. Falcon Mamba 7B will be released under TII Falcon Licence 2.0, the permissive Apache 2.0-based software licence, which includes an acceptable use policy that promotes the responsible use of AI. More information on the new model can be found at FalconLLM.TII.ae.