ThursdAI - Mar 20 - OpenAIs new voices, Mistral Small, NVIDIA GTC recap & Nemotron, new SOTA vision from Roboflow & more AI news · ThursdAI - The top AI news from the past week summary

Key takeaways

OpenAI has revolutionized voice technology with the release of new models, allowing for greater contextual understanding and emotion in generated speech.
Roboflow introduced the RF-DETR model and RF100-VL benchmark to address real-world applications in vision, illustrating the need for updated evaluation frameworks.
Mistral's return breaks the mold in open-source LLM development, outperforming established competitors and highlighting the industry's growth.
Google's efforts to enhance accessibility in AI tools contrast sharply with OpenAI's pricing strategy, raising questions about equity in access to advanced AI capabilities.
Tencent's advancements in 3D modeling signify a leap towards more dynamic and accurate content generation, paving the way for applications in animation and virtual reality.

Summary

Voice & Audio Revolution

This week's podcast was dominated by advancements in voice and audio technology, primarily driven by OpenAI's breakthroughs. They introduced new voice models, including GPT-4o mini TTS and GPT-4.0 transcribe models, which leverage a transformer-based architecture and are faster and cheaper than previous iterations, boasting state-of-the-art accuracy in multiple languages. Additionally, Kwindla Cramer from Daily highlighted the increased promptability of these new models, allowing for greater context in transcription tasks, such as accurately identifying dog breeds. The introduction of Semantic VAD (Voice Activity Detection) aims to improve AI agents' responsiveness by making turn detection more human-like. The voice segment also featured Canopy Labs' Orpheus 3B, another significant open-source release, praised for its natural-sounding speech and zero-shot voice cloning capabilities.

Vision & Video Innovations

Roboflow made a notable entrance in the vision segment with their new model RF-DETR, which is a real-time transformer-based object detection model beating the 60 mAP mark on the outdated COCO benchmark. The new benchmark, RF100-VL, was introduced to evaluate the real-world capabilities of vision-language models, revealing significant gaps in current performance—highlighting a mere 5.8% for top models. StepFun also showcased their image-to-video model TI2V, emphasizing its capabilities in generating high-quality animations from image prompts.

Open Source LLM Developments

Mistral returned to the open-source scene with their Mistral Small 3.1 model, outperforming Google’s Gemma 3, while LG surprised the community with their EXAONE Deep 32B model, a reasoning-capable AI linked to their appliances. NVIDIA released the Llama-Nemotron models with advanced reasoning capabilities and a focus on commercial use. Additionally, ByteDance introduced DAPO, a new reinforcement learning method aiming to improve efficiency in RL training processes.

Big CO LLMs and API Updates

In the realm of big tech, Google made a strategic move by making their DeepResearch tool free to users and enhancing their Gemini capabilities, promoting accessibility. In contrast, OpenAI's announcement of their O1 Pro API, priced at $600 per million output tokens, sparked concerns about the sustainability of its service primarily benefiting a wealthy few. NVIDIA's GTC conference showcased groundbreaking hardware advancements, with Jensen Huang reinforcing NVIDIA's commitment to shaping the AI future.

3D Innovations from Tencent

Tencent launched their updated Hunyuan 3D model which utilizes multi-view input to significantly enhance the quality and accuracy of three-dimensional modeling. This technology allows for near real-time generation of 3D models based on a single image or multiple perspectives. The developments point towards an exciting future where AI-generated 3D content could become commonplace.

Show notes

Hey, it's Alex, coming to you fresh off another live recording of ThursdAI, and what an incredible one it's been!

I was hoping that this week will be chill with the releases, because of NVIDIA's GTC conference, but no, the AI world doesn't stop, and if you blinked this week, you may have missed 2 or 10 major things that happened.

From Mistral coming back to OSS with the amazing Mistral Small 3.1 (beating Gemma from last week!) to OpenAI dropping a new voice generation model, and 2! new whisper killer ASR models with a Breaking News during our live show (there's a reason we're called ThursdAI) which we watched together and then dissected with Kwindla, our amazing AI VOICE and real time expert.

Not to mention that we also had dedicated breaking news from friend of the pod Joseph Nelson, that came on the show to announce a SOTA vision model from Roboflow + a new benchmark on which even the top VL models get around 6%! There's also a bunch of other OSS, a SOTA 3d model from Tencent and more!

And last but not least, Yam is back 🎉 So... buckle up and let's dive in. As always, TL;DR and show notes at the end, and here's the YT live version. (While you're there, please hit subscribe and help me hit that 1K subs on YT 🙏 )

Voice & Audio: OpenAI's Voice Revolution and the Open Source Echo

Hold the phone, everyone, because this week belonged to Voice & Audio! Seriously, if you weren't paying attention to the voice space, you missed a seismic shift, courtesy of OpenAI and some serious open-source contenders.

OpenAI's New Voice Models - Whisper Gets an Upgrade, TTS Gets Emotional!

OpenAI dropped a suite of next-gen audio models: gpt-4o-mini-tts-latest (text-to-speech) and GPT 4.0 Transcribe and GPT 4.0 Mini Transcribe (speech-to-text), all built upon their powerful transformer architecture.

To unpack this voice revolution, we welcomed back Kwindla Cramer from Daily, the voice AI whisperer himself. The headline news? The new speech-to-text models are not just incremental improvements; they’re a whole new ballgame. As OpenAI’s Shenyi explained, "Our new generation model is based on our large speech model. This means this new model has been trained on trillions of audio tokens." They're faster, cheaper (Mini Transcribe is half price of Whisper!), and boast state-of-the-art accuracy across multiple languages. But the real kicker? They're promptable!

"This basically opens up a whole field of prompt engineering for these models, which is crazy," I exclaimed, my mind officially blown. Imagine prompting your transcription model with context – telling it you're discussing dog breeds, and suddenly, its accuracy for breed names skyrockets. That's the power of promptable ASR! I recorded a live reaction aftder dropping of stream, and I was really impressed with how I can get the models to pronounce ThursdAI by just... asking!

But the voice magic doesn't stop there. GPT 4.0 Mini TTS, the new text-to-speech model, can now be prompted for… emotions! "You can prompt to be emotional. You can ask it to do some stuff. You can prompt the character a voice," OpenAI even demoed a "Mad Scientist" voice! Captain Ryland voice, anyone? This is a huge leap forward in TTS, making AI voices sound… well, more human.

But wait, there’s more! Semantic VAD! Semantic Voice Activity Detection, as OpenAI explained, "chunks the audio up based on when the model thinks The user's actually finished speaking." It’s about understanding the meaning of speech, not just detecting silence. Kwindla hailed it as "a big step forward," finally addressing the age-old problem of AI agents interrupting you mid-thought. No more robotic impatience!

OpenAI also threw in noise reduction and conversation item retrieval, making these new voice models production-ready powerhouses. This isn't just an update; it's a voice AI revolution, folks.

They also built a super nice website to test out the new models with openai.fm !

Canopy Labs' Orpheus 3B - Open Source Voice Steps Up

But hold on, the open-source voice community isn't about to be outshone! Canopy Labs dropped Orpheus 3B, a "natural sounding speech language model" with open-source spirit.

Orpheus, available in multiple sizes (3B, 1B, 500M, 150M), boasts zero-shot voice cloning and a glorious Apache 2 license. Wolfram noted its current lack of multilingual support, but remained enthusiastic, I played with them a bit and they do sound quite awesome, but I wasn't able to finetune them on my own voice due to "CUDA OUT OF MEMORY" alas

I did a live reaction recording for this model on X

NVIDIA Canary - Open Source Speech Recognition Enters the Race

Speaking of open source, NVIDIA surprised us with Canary, a speech recognition and translation model. "NVIDIA open sourced Canary, which is a 1 billion parameter and 180 million parameter speech recognition and translation, so basically like whisper competitor," I summarized. Canary is tiny, fast, and CC-BY licensed, allowing commercial use. It even snagged second place on the Hugging Face speech recognition leaderboard! Open source ASR just got a whole lot more interesting.

Of course, this won't get to the level of the new SOTA ASR OpenAI just dropped, but this can run locally and allows commercial use on edge devices!

ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Vision & Video: Roboflow's Visionary Model and Video Generation Gets Moving

After the voice-apalooza, let's switch gears to the visual world, where Vision & Video delivered some knockout blows, spearheaded by Roboflow and StepFun.

Roboflow's RF-DETR and RF100-VL - A New Vision SOTA Emerges

Roboflow stole the vision spotlight this week with their RF-DETR model and the groundbreaking RF100-VL benchmark. We were lucky enough to have Joseph Nelson, Roboflow CEO, join the show again and give us the breaking news (they published the Github 11 minutes before he came on!)

RF-DETR is Roboflow's first in-house model, a real-time object detection transformer that's rewriting the rulebook. "We've actually never released a model that we've developed. And so this is the first time where we've taken a lot of those learnings and put that into a model," Joseph revealed.

And what a model it is! RF-DETR is not just fast; it's SOTA on real-world datasets and surpasses the 60 mAP barrier on COCO. But Joseph dropped a truth bomb: COCO is outdated. "The benchmark that everyone uses is, the COCO benchmark… hasn't been updated since 2017, but models have continued to get really, really, really good. And so they're saturated the COCO benchmark," he explained.

Enter RF100-VL, Roboflow's revolutionary new benchmark, designed to evaluate vision-language models on real-world data. "We, introduced a benchmark that we call RF 100 vision language," Joseph announced. The results? Shockingly low zero-shot performance on real-world vision tasks, highlighting a major gap in current models. Joseph's quiz question about QwenVL 2.5's zero-shot performance on RF100-VL revealed a dismal 5.8% accuracy. "So we as a field have a long, long way to go before we have zero shot performance on real world context," Joseph concluded. RF100-VL is the new frontier for vision, and RF-DETR is leading the charge! Plus, it runs on edge devices and is Apache 2 licensed! Roboflow, you legends! Check out the RF-DETR Blog Post, the RF-DETR Github, and the RF100-VL Benchmark for more details!

StepFun's Image-to-Video TI2V - Animating Images with Style

Stepping into the video arena, StepFun released their image2video model, TI2V. TI2V boasts impressive motion controls and generates high-quality videos from images and text prompts, especially excelling in anime-style video generation. Dive into the TI2V HuggingFace Space and TI2V Github to explore further.

Open Source LLMs: Mistral's Triumphant Return, LG's Fridge LLM, NVIDIA's Nemotron, and ByteDance's RL Boost

Let's circle back to our beloved Open Source LLMs, where thi…

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sub.thursdai.news/subscribe

ThursdAI - Mar 20 - OpenAIs new voices, Mistral Small, NVIDIA GTC recap & Nemotron, new SOTA vision from Roboflow & more AI news