📅 ThursdAI - Feb 29 - Leap Year Special ✨

Happy leap year day everyone, very excited to bring you a special once-in-a-4 year edition of ThursdAI 👏

(Today is also Dune 2 day (am going to see the movie right after I write these here words) and well.. to some folks, this is the bull market ₿ days as well. So congrats to all who weathered the bear market!)

This week we had another great show, with many updates, and a deep dive, and again, I was able to cover most of the news AND bring you a little bit of a deep dive into a very interesting concept called Matryoshka Representation Learning (aka 🪆 embeddings) and two of the authors on paper to chat with me on the pod!

TL;DR of all topics covered:

* AI Art & Diffusion & 3D

* Playground releases a new diffusion foundational model Playground V2.5 (DEMO)

* Alibaba teasing EMO - incredible animating faces (example)

* Ideogram 1.0 announced - SOTA text generation (Annoucement)

* Open Source LLMs

* Gemma update - hard to finetune, not better than 7B mistral

* LLama 3 will release in June 2024, not anytime soon

* Starcoder 2 + stack V2 (Announcement)

* Berkeley Function-Calling leaderboard Leaderboard (Announcement)

* Argilla released OpenHermesPreferences the largest open dataset for RLHF & DPO (Announcement)

* STORM from Stanford to write long documents (Thread)

* Big CO LLMs + APIs

* Mistral releases Mistral Large & Le Chat (Announcement, Le Chat)

* Microsoft + Mistral strike a deal (Blog)

* Google teases GENIE - model makes images into interactive games (announcement)

* OpenAI allowing fine-tune on GPT 3.5

* Wordpress & Tumbler preparing to sell user data to OpenAI & Midjourney

* Other

* Mojo releases their MAX inference engine, compatible with PyTorch, Tensorflow & ONNX models (Announcement)

* Interview with MRL (Matryoshka Representation Learning) authors (in audio only)

AI Art & Diffusion

Ideogram 1.0 launches - superb text generation!

Ideogram, founded by ex google Imagen folks, which we reported on before, finally announces 1.0, and focuses on superb image generation. It's really great, and I generated a few owls already (don't ask, hooot) and I don't think I will stop. This is superb for meme creation, answering in multimedia, and is fast as well, I'm very pleased! They also announced a round investment from A16Z to go with their 1.0 release, definitely give them a try

Playground V2.5

Suhail Doshi and Playground release a new foundational image model called Playground v2.5 and it looks awesome, very realistic and honestly looks like it beats MJ and DALL-E on many simple prompts.

They also announced that this model received higher user preference scores based on 1K prompts (which we didn't get to see) but they have released this model into the wild, you can download it and play with a free demo provided by modal folks

Another SORA moment? Alibaba teases EMO 🤯 (website)

Ok this one has to be talked about, Alibaba released quite a few preview videos + paper about something called EMO, a way to animate a talking/singing Avatars from just 1 image. It broke my brain, and I couldn't stop staring at it. Honestly, it's quite quite something. This model animates not only the mouth, eyes are blinking, there are emotions, hairs move, even earrings, and the most impressive, the whole Larynx muscle structure seem to be animated as well!

Just look at this video, and then look at it again.

The Github repo was created but no code released and I really hope we get this code at some point, because animating videos with this fidelity + something like SORA can mean so many possible creations!

I wrote this tweet only two weeks ago, and I'm already feeling that it's outdated and we're farther along on the curve to there with EMO, what a great release!

And just because it's so mind-blowing, here are a few more EMO videos for you to enjoy:

Open Source LLMs

Starcoder 2 + The Stack V2

Folks at hugging face and BigCode have released a beast on us, StarCoder 2 ⭐️ The most complete open Code-LLM 🤖 StarCoder 2 is the next iteration for StarCoder and comes in 3 sizes, trained 600+ programming languages on over 4 Trillion tokens on Stack v2. It outperforms StarCoder 1 by margin and has the best overall performance across 5 benchmarks 🚀🤯.

TL;DR;🧮 3B, 7B & 15B parameter version🪟 16384 token context window🔠 Trained on 3-4T Tokens (depending on size)💭 600+ Programming languages🥇 15B model achieves 46% on HumanEval🧠 Grouped Query Attention and Sliding Window Attention💪🏻 Trained on 1024 x H100 NVIDIA GPUs✅ commercial-friendly license🧑🏻‍💻 Can be used for local Copilots

The Stack v2 is a massive (10x) upgrade on the previous stack dataset, containing 900B+ tokens 😮

Big CO LLMs + APIs

🔥 Mistral announces Mistral-Large + Le Chat + Microsoft partnership

Today, we are releasing Mistral Large, our latest model. Mistral Large is vastly superior to Mistral Medium, handles 32k tokens of context, and is natively fluent in English, French, Spanish, German, and Italian.

We have also updated Mistral Small on our API to a model that is significantly better (and faster) than Mixtral 8x7B.

Lastly, we are introducing Le Chat , a chat interface (currently in beta) on top of our models.

Two important notes here, one, they support function calling now on all mistral models in their API, which is a huge deal, and two, the updated Mistral Small to a "significantly better and faster" model than Mixtral 8x7B is quite the hint!

I want to also highlight Arthur’s tweet clarifying their commitment to Open Source because it's very important. They released a new website, it again had mentions of "don't train on our models" which they removed, and the new website also had removed the section that committed them to open weights and they put a much bigger section back up quickly!

This weeks Buzz (What I learned with WandB this week)

I mentioned this before, but this may shock new subscribers, ThursdAI isn't the only (nor the first!) podcast from Weights & Biases. Our CEO Lukas has a long standing podcast that's about to hit 100 episodes and this week he interviewed the CEO of Mayo Clinic - John Hamalka

It's a fascinating interview, specifically because Mayo Clinic just recently announced a mutli-year collaboration with Cerebras about bringing AI to everyone who googles their symptoms and ends up on mayo clinic websites anyway, and apparently John has been in AI for longer that I was alive so he's incredibly well positioned to do this and bring us the AI medicine future!

Modular announces MAX (Modular Accelerated Xecution) Developer Edition Preview (blog)

Modular, the company that created Mojo Lang from Chris Lattner, has now announced the second part of their stack, coming to all of us, and it's called MAX. It's an inference engine that has Mojo built in, that supports PyTorch, Tensorflow and ONNX and is supposedly going to run the same AI models we run now, significantly faster. MAX is a unified set of tools and libraries that unlock performance, programmability and portability for your AI inference pipelines

Right now they support only CPU inference, and significantly boost performance on CPU, however, they are planning GPU support soon as well, and promise up to 5x faster AI inference for most models like Mistral, LLama etc

I personally think this is a huge development, and while it's still early, definitely worth taking a look at the incredible speed performances that we are seeing lately, from Groq (as we chatted with them last week) and Modular, we're are very well on our way to run huge models faster, and small models instantly!

🪆 MRL (Matryoshka Embeddings) interview with Aditya & Prateek

Recently OpenAi has released 2 new embeddings models recently that replaced their ada-002 embeddings, and when they released it, they mentioned a new way of shortening dimensions. Soon after, on X, the authors of a 2022 paper MRL (Matryoshka Representation Learning) spoke out and said that this new "method" is actually MRL, the concept they came up with and presented at NeurIPS.

Since then I saw many folks explore Matryoshka embeddings, from Bo Wang to Connor Shorten and I w…