📅 ThursdAI - Mar 28 - 3 new MoEs (XXL, Medium and Small), Opus is 👑 of the arena, Hume is sounding emotional + How Tanishq and Paul turn brainwaves into SDXL images 🧠👁️

Hey everyone, this is Alex and can you believe that we're almost done with Q1 2024? March 2024 was kind of crazy of course, so I'm of course excited to see what April brings (besides Weights & Biases conference in SF called Fully Connected, which I encourage you to attend and say Hi to me and the team!)

This week we have tons of exciting stuff on the leaderboards, say hello to the new best AI in the world Opus (+ some other surprises), in the open source we had new MoEs (one from Mosaic/Databricks folks, which tops the open source game, one from AI21 called Jamba that shows that a transformers alternative/hybrid can actually scale) and tiny MoE from Alibaba, as well as an incredible Emotion TTS from Hume.

I also had the pleasure to finally sit down with friend of the pod Tanishq Abraham and Paul Scotti from MedArc and chatted about MindEye 2, how they teach AI to read minds using diffusion models 🤯🧠👁️

Thank you for reading ThursdAI - Recaps of the most high signal AI weekly spaces. This post is public so feel free to share it.

TL;DR of all topics covered:

* AI Leaderboard updates

* Claude Opus is number 1 LLM on arena (and in the world)

* Claude Haiku passes GPT4-0613

* 🔥 Starling 7B beta is the best Apache 2 model on LMsys, passing GPT3.5

* Open Source LLMs

* Databricks/Mosaic DBRX - a new top Open Access model (X, HF)

* 🔥 AI21 - Jamba 52B - Joint Attention Mamba MoE (Blog, HuggingFace)

* Alibaba - Qwen1.5-MoE-A2.7B (Announcement, HF)

* Starling - 7B that beats GPT3.5 on lmsys (HF)

* LISA beats LORA as the frontrunner PeFT (X, Paper)

* Mistral 0.2 Base released (Announcement)

* Big CO LLMs + APIs

* Emad leaves stability 🥺

* Apple rumors - Baidu, Gemini, Anthropic, who else? (X)

* This weeks buzz

* WandB Workshop in SF confirmed April 17 - LLM evaluations (sign up here)

* Vision & Video

* Sora showed some demos by actual artists, Air Head was great (Video)

* Tencent Aniportait - generate Photorealistic Animated avatars (X)

* MedArc - MindEye 2 - fMRI signals to diffusion models (X)

* Voice & Audio

* Hume demos EVI - empathic voice analysis & generation (X, demo)

* AI Art & Diffusion & 3D

* Adobe firefly adds structure reference and style transfer - (X, Demo)

* Discussion

* Deep dive into MindEye 2 with Tanishq & Paul from MedArc

* Is narrow finetuning done-for with larger context + cheaper prices - debate

🥇🥈🥉Leaderboards updates from LMSys (Arena)

This weeks updates to the LMsys arena are significant. (Reminder in LMsys they use a mix of MT-Bench, LLM as an evaluation and user ELO scores where users play with these models and choose which answer they prefer)

For the first time since the Lmsys arena launched, the top model is NOT GPT-4 based. It's now Claude's Opus, but that's not surprising if you used the model, what IS surprising is that Haiku, it's tiniest, fastest brother is now well positioned at number 6, beating a GPT4 version from the summer, Mistral Large and other models while being dirt cheap.

We also have an incredible show from the only Apache 2.0 licensed model in the top 15, Starling LM 7B beta, which is now 13th on the chart, with incredible finetune of a finetune (OpenChat) or Mistral 7B. 👏

Yes, you can now run a GPT3.5 beating model, on your mac, fully offline 👏 Incredible.

Open Source LLMs (Welcome to MoE's)

Mosaic/Databricks gave us DBRX 132B MoE - trained on 12T tokens (X, Blog, HF)

Absolutely crushing the previous records, Mosaic has released the top open access model (one you can download and run and finetune) in a while, beating LLama 70B, Grok-1 (314B) and pretty much every other non closed source model in the world not only on metrics and evals, but also on inference speed

It uses a Mixture of Experts (MoE) architecture with 16 experts that each activate for different tokens. this allows it to have 36 billion actively parameters compared to 13 billion for Mixtral. DBRX has strong capabilities in math, code, and natural language understanding.

The real kicker is the size, It was pre-trained on 12 trillion tokens of text and code with a maximum context length of 32,000 tokens, which is just incredible, considering that LLama 2 was just 2T tokens. And the funny thing is, they call this DBRX-medium 👀 Wonder what large is all about.

Graph credit Awni Hannun from MLX (Source)

You can play with the DBRX here and you'll see that it is SUPER fast, not sure what Databricks magic they did there, or how much money they spent (ballpark of ~$10M) but it's truly an awesome model to see in the open access! 👏

AI21 releases JAMBA - a hybrid Transformer + Mamba 58B MoE (Blog, HF)

Oh don't I love #BreakingNews on the show! Just a few moments before ThursdAI, AI21 dropped this bombshell of a model, which is not quite the best around (see above) but has a few very interesting things going for it.

First, it's a hybrid architecture model, capturing the best of Transformers and Mamba architectures, and achieving incredible performance on the larger context window size (Transformers hardware requirements scale quadratically with attention/context window)

AI21 are the first to show (and take the bet) that hybrid architecture models actually scale well, and are performant (this model comes close to Mixtral MoE on many benchmarks) while also being significantly cost advantageous and faster on inference on longer context window. In fact they claim that Jamba is the only model in its size class that fits up to 140K context on a single GPU!  

This is a massive effort and a very well received one, not only because this model is Apache 2.0 license (thank you AI21 👏) but also because this is now the longest context window model in the open weights (up to 256K) and we've yet to see the incredible amount of finetuning/optimizations that the open source community can do once they set their mind to it! (see Wing from Axolotl, add support for finetuning Jamba the same day it released)

Can't wait to see the benchmarks for this model once it's properly instruction fine-tuned.

Small MoE from Alibaba - Qwen 1.5 - MoE - A2.7B (Blog, HF)

What a week for Mixture of Experts models, we got an additional MoE from the awesome Qwen team, where they show that training a A2.7B (the full model is actually 14B but only 2.7B are activated at the same time) is cheaper, 75% reduction in training costs and 174% improvement in inference speed!

Also in open source:

Lisa beats LORA for the best parameter efficient training

📰 LISA is a new method for memory-efficient large language model fine-tuning presented in a Hugging Face paper💪 LISA achieves better performance than LoRA with less time on models up to 70B parameters🧠 Deep networks are better suited to LISA, providing more memory savings than shallow networks💾 Gradient checkpointing greatly benefits LISA by only storing gradients for unfrozen layers📈 LISA can fine-tune models with up to 7B parameters on a single 24GB GPU🚀 Code implementation in LMFlow is very simple, only requiring 2 lines of code🤔 LISA outperforms full parameter training in instruction following tasks

Big CO LLMs + APIs

Emad departs from Stability AI.

In a very surprising (perhaps unsurprising to some) move, Emad Mostaque, founder and ex-CEO of stability announces his departure, and focus on decentralized AI

For me personally (and I know countless others) we all started our love for Open Source AI with Stable Diffusion 1.4, downloading the weights, understanding that we can create AI on our machines, playing around with this. It wasn't easy, stability was sued to oblivion, I think LAION is still down from a lawsuit but we got tons of incredible Open Source from Stability, and tons of incredible people who work/worked there.

Big shoutout to Emad and very excited to see what he does next

Throwback to NEURIPS where Emad borrowed my GPU Poor hat and wore it ironically 😂 Promised me a stability hat but... I won't hold it against it him 🙂

This weeks Buzz (What I learned with WandB this week)

I'm so stoked about the workshop we're running before the annual Fully Connected conference in SF! Come hear about evaluations, better prompting with …