Last Week in AI - #200 - ChatGPT Roadmap, Musk OpenAI Bid, Model Tampering

#200 - ChatGPT Roadmap, Musk OpenAI Bid, Model Tampering

Feb 17, 2025 - 01h 48m 09s

Our 200th episode with a summary and discussion of last week's big AI news! Recorded on 02/14/2025

Join our brand new Discord here! https://discord.gg/nTyezGSKwP

Hosted by Andrey Kurenkov and Jeremie Harris. Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

In this episode:

OpenAI announces plans to unify their model offerings, moving away from multiple separate models (GPT-4, Claude, etc.) toward a single unified intelligence system, with free users getting "standard intelligence" and Plus subscribers accessing "higher intelligence" levels.
Adobe launches their Sora-rivaling AI video generator with 1080p output and 5-second clips, emphasizing production-ready content for films and introducing new pricing tiers through Firefly subscriptions at $10-30 per month.
Elon Musk and a consortium offer $97.4 billion to acquire OpenAI's nonprofit entity, potentially complicating the company's transition to a for-profit structure, though Sam Altman quickly dismissed the offer's viability.
TSMC implements stricter chip sales restrictions to China, requiring government-approved third-party packaging houses for chips using 16nm and below processes, aligning with US export control measures and affecting major tech companies like Nvidia and AMD.

Timestamps + Links:

(00:00:00) Intro / Banter
(00:01:25) Response to listener comments
(00:02:41) News Preview
Tools & Apps
- (00:03:58) Adobe’s Sora rivalling AI video generator is now available for everyone
- (00:09:45) OpenAI lays out plans for GPT-5
- (00:16:42) OpenAI is rethinking how AI models handle controversial topics
- (00:21:28) Perplexity AI launches new ultra-fast AI search model Sonar
- (00:23:45) YouTube AI updates include auto dubbing expansion, age ID tech, and more
Applications & Business
- (00:24:37) Musk-led group makes $97.4 billion bid for control of OpenAI
- (00:34:32) Anthropic’s next major AI model could arrive within weeks
- (00:39:09) AI chip startup Groq secures $1.5 billion commitment from Saudi Arabia
- (00:42:15) OpenAI reportedly planning to build its first AI chip in 2026
Projects & Open Source
Research & Advancements
- (00:58:24) Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
- (01:04:24) Distillation Scaling Laws
- (01:10:06) Matryoshka Quantization
- (01:17:47) How much AI compute exists globally? How rapidly is it growing?
Policy & Safety
- (01:21:29) US and UK refuse to sign summit declaration on AI safety
- (01:25:43) Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- (01:34:40) xAI Risk Management Framework (Draft)
- (01:39:59) TSMC bans more chip sales to China due to stricter U.S. export sanctions
(01:42:38) Listener requested topic
Synthetic Media & Art
- (01:43:48) Thomson Reuters Wins First Major AI Copyright Case in the US
- (01:44:46) Scarlett Johansson calls for deepfake ban after AI video goes viral
(01:45:55) Outro

Discover more and add this to your library.

Our 200th episode with a summary and discussion of last week's big AI news! Recorded on 02/14/2025

Join our brand new Discord here! https://discord.gg/nTyezGSKwP

Hosted by Andrey Kurenkov and Jeremie Harris. Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

In this episode:

OpenAI announces plans to unify their model offerings, moving away from multiple separate models (GPT-4, Claude, etc.) toward a single unified intelligence system, with free users getting "standard intelligence" and Plus subscribers accessing "higher intelligence" levels.
Adobe launches their Sora-rivaling AI video generator with 1080p output and 5-second clips, emphasizing production-ready content for films and introducing new pricing tiers through Firefly subscriptions at $10-30 per month.
Elon Musk and a consortium offer $97.4 billion to acquire OpenAI's nonprofit entity, potentially complicating the company's transition to a for-profit structure, though Sam Altman quickly dismissed the offer's viability.
TSMC implements stricter chip sales restrictions to China, requiring government-approved third-party packaging houses for chips using 16nm and below processes, aligning with US export control measures and affecting major tech companies like Nvidia and AMD.

Timestamps + Links:

(00:00:00) Intro / Banter
(00:01:25) Response to listener comments
(00:02:41) News Preview
Tools & Apps
- (00:03:58) Adobe’s Sora rivalling AI video generator is now available for everyone
- (00:09:45) OpenAI lays out plans for GPT-5
- (00:16:42) OpenAI is rethinking how AI models handle controversial topics
- (00:21:28) Perplexity AI launches new ultra-fast AI search model Sonar
- (00:23:45) YouTube AI updates include auto dubbing expansion, age ID tech, and more
Applications & Business
- (00:24:37) Musk-led group makes $97.4 billion bid for control of OpenAI
- (00:34:32) Anthropic’s next major AI model could arrive within weeks
- (00:39:09) AI chip startup Groq secures $1.5 billion commitment from Saudi Arabia
- (00:42:15) OpenAI reportedly planning to build its first AI chip in 2026
Projects & Open Source
Research & Advancements
- (00:58:24) Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
- (01:04:24) Distillation Scaling Laws
- (01:10:06) Matryoshka Quantization
- (01:17:47) How much AI compute exists globally? How rapidly is it growing?
Policy & Safety
- (01:21:29) US and UK refuse to sign summit declaration on AI safety
- (01:25:43) Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- (01:34:40) xAI Risk Management Framework (Draft)
- (01:39:59) TSMC bans more chip sales to China due to stricter U.S. export sanctions
(01:42:38) Listener requested topic
Synthetic Media & Art
- (01:43:48) Thomson Reuters Wins First Major AI Copyright Case in the US
- (01:44:46) Scarlett Johansson calls for deepfake ban after AI video goes viral
(01:45:55) Outro

Sponsors, Brands, and Products

Discord

Join our brand new Discord here!

AI-Generated

Highlights from the 200th Episode

The podcast celebrates its 200th episode, reflecting on its journey since its inception in 2020. The hosts, Andrik and Jeremy, discuss listener feedback and potential changes to the podcast format, including addressing listener-requested topics in future episodes.

OpenAI's Roadmap and Model Changes

OpenAI announces its plans to unify its model offerings, moving away from multiple models like GPT-4 and Claude, toward a singular intelligent system. Under the new structure, free users will receive a 'standard intelligence' while Plus subscribers will access higher intelligence levels.

Adobe's New AI Video Tool

Adobe has launched its Sora-rivaling AI video generator, capable of producing 1080p video clips lasting up to five seconds. The tool is available through a public beta and introduces new subscription tiers ranging from $10 to $30 per month, as part of the Firefly service.

Elon Musk's OpenAI Bid

Elon Musk presents a monumental $97.4 billion bid to acquire OpenAI's nonprofit entity, a move that complicates the company's ongoing transition to a for-profit structure. Sam Altman quickly dismisses the viability of Musk's offer.

Geopolitical Tensions and TSMC's Chip Sales Restrictions

TSMC enforces stricter sales restrictions for chips using 16nm processes or below for China, aligning with US export control measures. This affects major tech companies like Nvidia and AMD, highlighting ongoing tensions in the semiconductor industry.

Research and Development in AI Safety

A draft of xAI's risk management framework is released, detailing protocols for handling safety and ethical concerns related to AI. This follows previous discussions on utility engineering and the emergence of consistent value systems within AI models.

Legal Developments in AI and Copyright

The podcast covers significant legal cases around AI, including Thomson Reuters winning a major AI copyright lawsuit and Scarlett Johansson's call for a ban on deepfakes, showcasing ongoing discussions about the intersection of technology and intellectual property.

- The podcast celebrated its 200th episode, reflecting on its history and listener engagement.
- OpenAI is shifting towards a unified intelligence model, simplifying user experience.
- Adobe has launched a new AI video tool, emphasizing creative applications with a subscription model.
- Elon Musk's bid for OpenAI adds complexity to the organization's structure.
- TSMC's restrictions on chip sales to China illustrate the geopolitical ramifications in technology.
- xAI is prioritizing AI safety and governance with its new risk management framework.
- Recent legal cases highlight the evolving landscape of AI copyright and ethical considerations.

Transcription

We can hear a chat about what's going on with AI.

As usual in this episode, we will summarize and discuss some of last week's most interesting AI news.

And you can also go to last week in.ai for all the links for this episode and also for our text newsletter.

I am one of your hosts, Andrik Rankov.

My background is I've having studied AI in grad school and now working at NAI Startup.

And everybody, what's up?

My name is Jeremy Harris.

I'm your other co-host, Co-Foundry Ladsdon AI, an AI national security company.

You know that if you're listening to the podcast every week, it's almost get tiring to get the bios every time, but at least now you know we're, I guess, you guys don't know who we are.

I wonder if we'll maybe consider Returning your bios at every time.

I don't know how many new listeners we get to justify that and I guess we can go in and jump to Addressing the fact that this will be the 200th episode of the last week in AI.

There's a few more on the podcast feed you do some interviews and stuff, but this will be number 200.

damn I feel old I know let's quickly celebrate that for a bit musically there was also an Apple podcast review that just said don't ever stop so I guess so far pretty good 200 we've had a good record of doing this Jeremy you joined at episode 110 for reference okay so I'm still yeah I'm on episode 90.

Yeah, you are an episode 90.

For anyone who hasn't been a long-time listener, we were doing this back starting 2020.

We had a different co-host, Sharon.

I guess almost did stop actually in 2022.

We had a bit of a break and then thanks to Jeremy, we were able to come back and keep going.

So we'll see.

Hopefully we won't stop at this until age.

I guess here.

I think that's okay.

That gives us at least two weeks or so.

I know.

Given that, maybe a good time to make some changes.

Maybe we'll retire, we'll buy those, start an episode, and we'll also start discussing listener-requested topics towards the end of a show since we have gotten more of those kinds of things.

That'll be kind of the last section after all the news.

The only thing we'll be talking about as far as listeners is any corrections that we need to address.

So maybe that will be better for a lot of listeners quick preview of what we'll be talking about in this episode We have some cool new models in tools and apps Adobe is coming out with a new saw arrival and we have some Updates to kind of the timeline of LLMs Applications and business of course more opening I drama with Elon Musk got to talk about that and and some kind of previews of what's coming with other LLMs like Anthropic.

Got a few neat projects and open source stories with one that text-to-speech model and some data sets, research and advancement, similarly, talking about some distillation scaling laws and some tampering attacks for LLMs, which would be cool.

Pause here, safety, a bit of a mix of discussions between the US and UK, some geopolitical stuff, and paper also on utility engineering.

And we'll cap it off with a discussion of some updates to AI copyright law, which has actually happened at long last.

Yeah.

And then we will cap it off with a bit of discussion of listener requested topics.

So hopefully you won't go forever and we'll actually get to that before we are completely out of energy.

And, with that being said, let us get going in tools and apps.

Our first story is Adobe's Sora Riveling AI video generator is now available for everyone.

So Adobe has had a whole suite of generative models of their own starting with Firefly.

Now their video generator that they We call Generate Video is out for public beta access.

And it is available through the Firefly web app, which also has image generation and some translation stuff.

There are two ways you can use it, text to video and image to video.

So pretty similar to other things, videos are output and HD, 1080p, 24 frames per second, And you can generate them in 90 seconds or more, so it takes a minute and a half, two minutes to generate something and the length of those is up to five seconds.

So yeah, this is a preview, of course, and I guess it will see how long it takes to actually go out into their mainline tools.

They have been integrating various Generative models into Photoshop for instance with generative fill and the Dolby is marketing this generate video tool as a production ready Kind of tool for use in films about any copyright issues, which has also been there kind of line with all of their Generative models that this is something you can use when making a film and not worry about people suing you So, interesting to see if this actually does have a lag up against Sora and some other competitors.

Yeah, it really feels like we're reliving that kind of on-ramp, the generative image, like text-to-image on-ramp, that we felt through two years ago, two and a half years ago, over the course of about 18 months or so, you went from pretty slow, frustrating, unremarkable capabilities to like, you know, photorealistic stuff.

And now, from one release to the next, you can't really tell the difference.

I will keep eating this drum.

I think this is kind of a race to the bottom on pricing.

Ultimately, it's going to happen here as well, where everybody's just going to be charging essentially for the cost of inference plus some small margin.

This is really looking like it's going to be commoditized as well.

I don't really see an intrinsic reason why video would be any different from images, but just to give you a sense, still of the road map.

So the video output that they can provide here is in 1080p, 24 frames per second.

So decent into ish, maybe a little on the low end, but it isn't an upgrade from their original 720p quality.

So we're getting more and more into deeper HD here, interestingly, still choosing to stick with that five seconds, by contrast, Sora can do 20 seconds already.

So see this tradeoff, where do you put the compute?

Do you put the compute in generating longer videos that are logically coherent across longer periods of time, or do you invest your compute in higher resolution?

You're 4K, you're 1080p, that sort of thing.

By the way, Adobe does say they're also working on a faster, lower resolution, what they call an ideation model and a 4K model, which are both coming soon, supposedly.

But this is, I guess, an interesting use case and a bit of a bifurcation that at least, and I'm sure there are things like this in-text to image.

You're going to have to excuse me because I don't know that space quite as well, other than what we talk about on the podcast.

But there's always, in every domain, we get deep enough, there are interesting trade-offs.

But with video, it's extra obvious, right?

this idea of like, do I go higher resolution, do I put my computer into higher resolution frames, or into coherence across a larger number of frames?

You can see why OpenAI might have gone for the coherence side more than Adobe with these 22nd videos, just because coherence is at least intuitively a bit closer to the sort of reasoning that you need to make AGI, right, you have to be able to track objects in their interactions across longer periods of time, that kind of starts to look more world model-like than necessarily, you know, high-resolution image.

Though, again, you know, there's some truth to both of them.

I just intuitively think maybe the longer-time resolution of opening eyes Sora is a little bit more in tune with that.

So maybe an interesting niche for them to carve out from themselves here at Adobe, focusing on, you know, high-resolution video quality 4K and all that.

Right.

And I think we did discuss this preview they posted some videos of it prior to it will be available so they are trying to differentiate in a couple other ways they do say that for instance you can prompt the model with camera controls as one of the kind of common things things like zoom out and give me this particular shot and on my end I will keep being a drum but I think image to video is a much bigger deal than text to video in general, as kind of an application of special effects, of, you know, removing weird artifacts in your video, all sorts of practical applications I can see being used.

So wouldn't be surprised if they're also emphasizing that.

And also worth covering in addition to what's coming out, they have introduced new subscription tiers for Firefly with credits for these model.

So there's a Firefly standard subscription, which costs 10 dollars per month, gives you 200 video or audio credits.

Not to sure how much that converts to seems like maybe 25 second video generations.

So not a ton.

And there is the pro subscription, which is $30 a month, which is 7,000 credits.

So about 75 second videos.

So starting to commercialize it, I wonder how this will connect to their existing subscription tiers for their tool use for things like the editing suite.

Next story and moving on next we have OpenAI and them saying what will get to GP5 as opposed to anything that they are releasing so Sam Altman posted on Twitter slash X where he had not a super detailed but a fairly specific idea of a roadmap they're looking at and the highlight is that they are trying to move away from the current paradigm where you have like six models choose from, you know, have GP40, you have all one preview, I think still, or free mini now, and if you use chat GPT you have this model drop down where you need to select which model you want to use for any given task and you have you know various amounts of token usage per model.

So it all gets a little bit cumbersome as a specifically new user as well.

So what they are saying is that the new series and models will be sort of unifying this O series of using models and the GPT series of models where the model will just sort of know what to do based on your prompt.

And so we won't be seeing presumably, oh, free, we won't be seeing anything until we get this kind of unified tool.

And apparently, when it does come out, free chat GPT users will have a limited access at what you're calling the standard intelligence level.

And plus, subscribe, you'll have a higher intelligence level.

So yeah, it seems like they're going to change up for user experience, which is interesting, given it, But it hasn't really happened since kind of early days of chatbots.

Yeah, it's been good inspiration for me too.

Anytime I offer any consulting work, I now have the two tiers I offer them the standard intelligence level and then the high intelligence level.

That's a good business model, okay?

It really is, it really is.

This podcast, if you're wondering, is high.

This is as high as I can go.

So I don't bother.

But anyway, so one interesting update too, is we didn't know that this was gonna happen, right?

If the big update here is the bundling, essentially, the, as Sam says, somewhere in his tweet, right?

He's like, we hate the model picker just as much as you do, and want to return to magic unified intelligence, right?

They want this feeling to be that you show up at the interface, the console or whatever, you pump in your query.

You just kind of get the answer from the model that's most appropriate for your query.

And that does make sense.

We did get, I think a couple of weeks ago, Kevin Wilde, OpenAI's CPO, did say that they were on track to release 03 in the kind of February, March timeframe.

So this is still consistent if you kind of include it in that roll up though, Sam maybe gave himself a bit of oil room.

He gave this ambiguous timeline of when these things would roll out saying it'd be weeks slash months for GPT-5 at least.

So there is a little bit of ambiguity, but as you said, I like what you said, not high detail, but high specificity that does seem like we're getting here.

We're finally getting named releases.

One thing to keep in mind, right?

So GPD 4.5, Sam Altman is now saying that is going to be the model internally that was codenamed Orion.

We saw a big stink about that for a long time, right?

The strawberry project, which turned into 01, and then Orion, which is turning into GPD 4.5. And we've had this conversation on podcasts before.

A lot of people talk about, you know, what will GPD 5 be able to do?

What will GPD 4.5 be able to do?

And it's really kind of, in a sense, pointless to talk about things in quite that way.

Some people use that colloquially to mean like really the next order of magnitude or two ooms of scale, which makes total sense.

But for others, there's this kind of fixation on just like the incrementing of the name.

And that's just something to keep in mind.

OpenAI internally is building a crap ton of different models.

They don't themselves know which one they're going to call what.

And so for a long time, this thing was just called Orion.

And that was just the model under development.

And if it seemed like the vibes were right, they would release it as GPT 4.5, which happened here, but that hasn't, you know, that wasn't locked in initially.

So just to kind of clarify there, you know, if you're looking at speculation on GPT 4.5, whatever, the ooms are the thing that matter a lot more.

The actual orders of magnitude of compute that's thrown at these models, the names not so much.

But in any case, at least now we have a name to hang on to was previously the sort of secretive Orion based model project.

And by the way, that is a base model.

So, GPD 4.5 is going to be the last non-chain-of-thought model that OpenAI ships.

From now on, apparently, they're all going to be focused on, well, chain-of-thought is a loosely defined thing, but they're going to be focused on models that use the kind of O3-01 reasoning paradigm.

That's at least going to be the default.

And they will still have these models like GPD 4.5 that can one-shot, but the goal is to kind of orient more and more towards reasoning.

Right.

And to preview a bit, we will be talking about some similar sounding plans on FRAPIC down in the business section because it was some business use.

But the idea of unifying reasoning models of non-reasoning models seems like it's kind of up in the air where there's no clear need to separate, right?

You can just enable a model to be able to reason if it wants or thinks it's warranted or not to reason and just to output without this long trace, which actually does seem like a bit of a no-brainer when you think about it like why do you need to separate these out and it seems like a future maybe that you don't that you have a single model capable of both paradigms and as to the naming I do think it's a bit interesting to just take a quick look back, chat GPT when it first came out late 2022 was built on top of GPD 3.5 at the time GP4 came out in March of 2023 and then since then we've gotten into the O model family which was Omni because they're added things like audio to it GP4O and now we have O1 as their reasoning model so they did basically stop incrementing in favor of having new types of model names to reflect new capabilities and new types of model from GP4 where it used to be GP2, 3, few point five, four.

As you said, that was always just going up and scale and improving on your intelligence levels.

And since then, since mid-2023, we've seen kind of more focus on broadening what models are capable of in terms of inputs, outputs, and I guess the way they do reasoning.

Next story.

Next up, we have another bit of news from OpenAI.

Apparently, they're rethinking how AI models will handle controversial topics.

So this is part of a released 60-page model spec, which is their guidelines for AI model behavior.

And this update aspect does include details as to how to handle these controversial topics.

And seemingly they will make it so VAI is able to do some slightly more, I don't know, spicy things.

So for instance, there would be a variability to allow certain adult content in appropriate context.

And there may also be variability for users to enable disabled safety guardrails to customize that a bit.

So, yeah, again, not perhaps too surprising.

I think many people using it alums do complain or criticize when models are overly restricted and overly careful.

So, them moving away a bit could be a reasonable move.

Yeah, the model spec also is something we've covered before, but just if you're new to the concept.

In some ways, it sounds like a document that might be fine-tuned on, maybe just thrown at an LLM, and fine-tuned it on this and that tweaks its behavior.

Some truth to that, but mostly it's a meta-instruction document.

It's not just something that you load into a model's context or just uses fine-tuning, but generally describes the approach opening eye takes and that it recommends developers take at every level of the stack from data collection to training to serving the model to using it in applications.

And so it's sort of like their overall meta instructions to themselves and to others.

It is a lot more kind of detailed than it was previously.

And as you said, there's a lot of emphasis on what they call intellectual freedom here.

Sort of interesting, one might be tempted to suspect that this is sort of a response to the change in administration that, you know, previously they were perfectly happy, Sam Altman having been like kind of a long-time Democrat.

They were perfectly happy to kind of align with the sort of, you know, more, I guess, the more kind of woke orientation of the chap-off as it had been previously, now kind of switching over, which is a very Sam Altman thing to do, kind of go with the winds of change.

And that's sort of what you're getting here.

It is useful, and I think a lot of people, as you said, sort of getting frustrated with dealing with the model that was kind of overly constrained previously.

So yeah, I mean, there are a whole bunch of ways in which this thing gets used.

It's almost like pointless to go through them all, but one sort of sub-aspect here is there's like this chain of command that's outlined in the model spec that describes how the models are meant to prioritize high as different levels of instruction, right?

You can think of like their platform level rules that OpenAI uses to override everything else.

Like these are like safety safeguards or legal compliance things.

But then you've got like a system prompt.

So how do you rank the system prompt and importance relative to the prompts that the developer puts in?

And they've got all kinds of measures and we cover to paper that deals with this as well.

It also contains instructions that they give to human laborers during reinforcement learning for human feedback data collection.

And so the model spec really is a very all-encompassing document in that sense.

And yeah, there's a big blog post that comes along with it, where they're celebrating the higher adherence of the 01 series, relative to the GPT-4O series, to their spec.

So they see across all these things that they care about, that they have these funny sounding sort of category names for the things they're trying to quantify and care about.

So adherence to the chain of command that we just talked about, But staying in bounds, so essentially giving responses that are consistent with the prompts they've been given, seeking the truth together is another one sort of interesting, apparently a big improvement there, of O1 relative to 40, do the best work, so some notion of optimization for the actual correctness of the answer, use appropriate style.

So on all those UC improvements, you actually see a worsening of performance, though, on be approachable.

So 01, apparently less approachable than 40, that at least tracks my experience.

But anyway, kind of interesting, you sort of see it all laid out there.

So they're letting us know as well, they won't be publishing more blog posts with every update to the model spec.

But they do have this log that you can track, which is great for transparency, right?

You go check it, see how they've updated their meta instructions.

Next story.

And next we have a perplexity AI announcing a new ultra-fast model.

So there you have this new version of our own in-house model Sonar, which is built on top of Lama 3.37B.

And they're basically saying that this is much better than our other off-the-shelf models, like Cloud 3.5, Haiku, or GP4O Mini, for every particular thing that that perplexity does, which is AI-enabled web search.

It's one of the ways similar to GPD web search, to Gemini web search, you enter a query, it looks up some websites and the AI and leads those websites and either answers your question or summarizes or whatever.

So this one's owner is fine tuned for improved factual accuracy and readability and they are claiming that it can run at 1,200 tokens per second by using a cerebrus hardware.

They are way for scale engines.

So yeah, it's it's similar to ChatGit PT, I guess, in perplexity you do have a drop-down of what model you're using.

So if you are pro subscriber, you are paying user, you can enable this model.

Well, it's interesting to see Cerebris getting picked up in this way for this sort of application.

It's sort of like two challengers, right?

In the space, the search challenger in perplexity, and then the hardware challenger in Cerebris, the design challenger, kind of cool.

Yeah, the advantage of going with the kind of Cerebris for the search-focused functionality, you can imagine like lower latency.

One of the things that the wafer scale engine does is it gets rid of the need for data to jump between different chips basically, which reduces latency.

And you have advantages in memory access and some parallelism things too, potentially.

So I mean, it'd be interesting to kind of unpack more of this choice, but there are reasons that it does make quite a bit of sense that maybe we can go into in a hardware, so 2.0. Whenever that happens, whenever that happens.

And one last story for the section.

We have some updates on features coming to YouTube.

So there was a yearly letter from the CEO of YouTube And there were some small announcements that for instance, there is an audio auto dubbing feature translating your video with audio and other languages automatically with AI as a YouTube creator or someone who posts videos you will now be able to use it seemingly the within the end of this month and they are saying they'll also invest in tools for detecting and managing AI generated content like being able to detect AI likenesses that are being generated which as we'll get to later perhaps some actors and other people will appreciate.

And we're going to applications and business as promised we have some more drama between Elon mask at OpenAI.

This seems to be like every week kind of situation.

The latest development here is that consortium led by Elon Musk has made an offer of 97.4 billion to acquire the nonprofit entity controlling OpenAI.

So this is a bit technical, you could say, as OpenAI is trying to go for profit.

They need to tackle their kind of complicated structure where the for-profit arm of OpenAI, at least I think this is how it works, is controlled by this nonprofit entity.

And so the for-profit arm needs to in some sense buy the nonprofit entity to then be able to transition to for-profit.

So, in this case, there's a competing bid, essentially, for that non-profit that is in charge of the for-profit.

Some outman, as you might expect, quickly stated that this is not going to happen.

Made a funny little snarky tweet actually offering to buy Twitter for less than $10 billion.

But it does seem to be like the latest tactic to at least delay and make it tougher to be able to do this because this is a pretty high bid.

It's more than the for-profit part of OpenAI is likely to pay themselves, seemingly they're offering like 40 billion.

So it could be another headache.

Yeah, and this is one where, you know, the I'm not a lawyer caveat is so important.

but from the analysis I've seen, it doesn't seem quite interesting, so yeah, you alluded to it.

There's this sort of for-profit entity that's trying to break itself free of the shackles of the non-profit, right?

But the problem is that the non-profit was essentially, like, it does have complete control of the for-profit, which means that whatever value is contained in the for-profit, the non-profit has that and then some, right?

So how does a for-profit, that it's almost like a slave buying its own master, buying their own master, like that's sort of roughly what's going on here, and it's not clear that that even makes sense because theoretically, the slave is the master's property, I'm thinking in ancient Rome here, so how do you even work that out?

This is a very mangled analogy, but that's roughly the picture here.

So for Elon to come in, one of the key questions is, Sam wants to sell the for-profit to the nonprofit.

profit.

He wants to sell it apparently roughly $40 billion.

But for profit has to pay fair market value for the non-profit.

And if there's a competing bid, a concrete competing bid, that's more than double, then that suggests that, hey, the fair market value is actually quite a bit higher.

And it's not obvious that Sam Altman can even put together the amount of money that he'd need to compete with that bid from Elon.

So, you know, like you don't to scrounge together $100 billion in liquid capital to make an investment like this.

In fact, he struggled to raise the $100 billion that they have raised so far for Stargate.

And that was with a clear kind of build investment that they were already ending towards.

So the other thing that I've seen argued is if opening eye succeeds in their mission of building transformative, like general artificial general intelligence, then surely the value of the company, even the expectation value of the company, the nonprofit is way, way, way, way higher than $40 billion, and that seems super, super reasonable as an assessment.

And so there's, I mean, again, not a lawyer, but this seems to argue pretty strongly in favor of some kind of consideration for the fair market value being a lot higher than the $40 billion that essentially Sam Altman is trying to sell to himself because he's on both boards.

That's the other thing.

There's like this kind of conflict of interest thing going on where he He wants to sell to himself and know what a surprise, the sale price is like a lot cheaper than what I think a reasonable observer would assess.

But all kinds of legal nuances and intricacies that I'm certainly not following.

There are all kinds of extra layers to this.

So Larry Summers, he's one of the directors on the opening IP board, says he has not received any formal outreach from Elon Musk and the consortium of investors here related to this offer.

But this is weird because Elon's attorney also says that he did submit the bid to open a eyes board on Monday.

So I don't know how to square the circle.

I mean, not receive formal outreach from the consortium.

Is that maybe slightly different from, I mean, surely it's not, but slightly different from getting a letter from his lawyer.

I mean, surely he would qualify.

I don't know.

This is all a big thorny hair ball.

But the consortium is also quite interesting, right?

The consortium that is looking to invest along with Elon in this acquisition, it does include XAI itself, which the idea is it could maybe merge with OpenAI following a deal, which is kind of hilarious because OpenAI theoretically has a merge and assist clause in their mission charter thing, whatever, what do they say?

If another competing project that's value aligned seems like it's within about two years or so of making AGI, then we will join with them so that we don't have a late stage kind of competitive race to make this potentially very dangerous technology.

It would be really funny and I don't think somehow I don't think this is how it's going to shake out but it'd be really funny if they were sort of forced to merge and assist with XAI except because an offer came in from XAI and not out of their own volition.

I don't expect that to happen by the way.

My guess is that you know this somehow just doesn't and materialize, but hey, it's 2025.

Weird shit happens.

So one last thing I'll mention, there are a bunch of other interesting investors joining XAI in the bid, joining Elon in the bid.

So you've got Baron Capital, Valor Equity Partners.

You've got 8VC, which is a venture firm that's led by one of the co-founders of Palantir, Joe Lonstale.

So it's kind of interesting, a lot of movement here, and we'll see.

Like again, where I really wish I was a lawyer.

So I could understand what the actual implications of this are and how much it ties up open AI at the stage But it's it's definitely a distraction for all concerned.

That's that's for damn sure Right and for context right this is coming you're pretty soon after previous I guess attack the Simone nature where our lawsuits also going on saying that opening I shouldn't be able to go for a profit in the first place because they were set up as a non-profit.

That was, again, and lost you by Elon Musk.

Here, I'm also not a lawyer, but I give a little detail that I can, which is typically in corporations, they are shareholders and they are shareholders in the non-profit, the companies that have invested like Microsoft, and the board of directors of a given company has a fiduciary duty to shareholders, where they are meant to be making decisions in the best interest of them.

So, as far as I understand it, again, don't trust me and this, but if you are selling your company, you do have somewhat legal obligation to try and get the best price possible.

So that is one reason where this could be a pickle.

Aside from the fact that the board of directors of nonprofit could just want to get a bigger price.

And as you said, lots of details to go through like the nonprofit board was overhauled since Ottoman was pushed out as CEO in late 2023.

free, so now it's kind of an ottoman friendly board.

All sorts of drama, I guess, to get into.

Yeah, I do think so, again, not a lawyer, but I do think that the wiggle room that exists for Sam right now is in part from the fact that the nonprofit does not have a fiduciary obligation to its shareholders.

So as I understand it, it may be a different legal responsibility they have as a non-profit to like sort of basically you can't fraud.

You can't just raise a bunch of money as a non-profit and then sell out to a for-profit and effectively become a de facto for-profit.

The assets that the non-profit has accumulated, it must keep or it must keep the equivalent value.

And that's where I mean, that's such a good point.

I forgot to have a non-profit is actually a non-profit.

No, but, well, but it's like, basically a nonprofit, right, which is worth considering because that does have implications, right?

Yeah.

I mean, like, honestly, my head hurts right now, but like you're right, and anyway, so at the end of the day, my understanding is part of the argument hinges on, no, like $40 billion is fair compensation for the nonprofit.

Like there is some argument like that they have to make.

That is not even cool.

I mean, look, you've got open AIs about to raise whatever it is at like $300 billion dollar valuation from soft bank, right?

That's the latest rumor.

So if that's anywhere close to the case, 40 billion dollars is a laughable kind of valuation here for the acquisition.

So I don't know how that gets factored in either.

That seems to undermine the Sands claim too.

But again, if he's going ahead with this, surely he wouldn't make such a move without, like, and leave himself this open to this kind of law fair.

I really don't know.

But we'll see.

We need a lawyer on this podcast.

Next story.

And the next story is about one of the other competitors of OpenAI on Frapplic.

And this is some sort of insider information.

So not official communications from on Frapplic, but there was a report from the information which posts a lot of kind of internal stories that aren't official disclosed.

So this report, this goes that on FRAPIC is working on an upcoming model that is a hybrid model.

It can switch between two modes of deep reasoning and fast responses.

This does track in an interview on Monday, we see on FRAPIC did say that they are generally focused on trying to make their reasoning models differentiated.

So, and they, yeah, he did say literally that anthropic is puzzled by the idea that there are normal models and there are reasoning models and they're sort of different from each other.

So it seems that they will make this unified model and there will reportedly be a sliding scale alongside the model to allow developers to control costs.

Because, of course, part of the idea of reasoning models is they do more reasoning, they do more thinking, which leads to more tokens, which leads to higher cost.

So that is one of the implications of using such a model.

So as I kind of previewed, tracks very much with the stated roadmap for OpenAI by Altman.

Yeah, it's still no word on pricing.

One of the things that is missing, or that seems to be missing right now, is models that have an intuition for when they should apply more compute to a problem, kind of within themselves.

So rather than having an externally tunable, you know, dial that allows you to say, okay, put in this, this many flops or this many tokens on average in your response, like having the model actually figured that out for itself, I don't know that that's fully separable from the intent of the prompt or sometimes you know that like you want to ask a question at different levels of detail, but there is a dimension of this where some of that could be offloaded to the system that doesn't seem like it's this.

I'm just kind of noting that that's something that we still haven't seen get dug up yet.

But in any case, kind of interesting, this would make Anthropic first out the gate with this capability, apparently one of the things this information piece says is that when the The anthropic model is allowed to think for the maximum amount of time when you tune that dial.

In other words, all the way to the right, it does apparently outperform the O3 mini model set on high.

That would make it, nominally, if nothing changes in the next couple hours, the most performant model that is released, they are focused more on the enterprise market.

This is the speculation and the article is like, okay, maybe that's part of the reason why they're focusing more on this dial.

It's putting more effort into features that give developers more control over the cost, the speed, you know, the pricing through the sliding scale approach rather than, you know, opening eye has these kind of three settings.

You can go low, medium, high and some people feel that it's hard to predict how many tokens the model is actually going to process with each of those, you know, at each of those levels.

And so it's difficult to predict how expensive your query is going to be, you know, here you have an anthropic kind of leaning into giving you a bit more control over that.

Last little bit of detail that was in this, in this report, apparently, the differentiator now for Anthropic remains the understanding of complex code bases.

So one of the things that we've consistently seen and that I've experienced at least is like, you know, Cloud 3.5 saw it and you just like really, really get a coding better in some ways, in some contexts than like, than any of the kind of open AI products that you can use anyway at a comparable price point.

So this is apparently going to be going to persist, especially they're orienting towards large complex code basis.

So really moving towards automation to fall on software engineering, where you're looking at thousands of files making complete lines of code that work just the first time.

Whereas opening eye is better as they put it more academic problems like competitive programming, which can reflect a little bit of metric hacking too, because there's so much more benchmarks and metrics that focus on like competitive programming than necessarily just the kind of like, well, sweet benchmark.

Actually, actual programming, right?

Yeah, exactly.

Yeah, yeah.

Next story.

Next, we have AI chip startup GROC has secured 1.5 billion in a commitment to invest from Saudi Arabia.

So Groc, that's Groc with a Q, not Groc with a K, is a leading competitor to Nvidia as a hardware provider.

They have their own kind of customized hardware solution, also similar to cerebrus, and they are have been added for a while.

We covered them getting some funding like October-ish of last year they had a series D-round with 640 million won't yet.

Now they have this 1.5 billion investment from Saudi Arabia which is tracking with a bunch of patterns of them being to raise quite a lot of funds they are currently or they have been valued at 2.8 billion since August and also Saudi Arabia being a major investor, as I think you've said last episode, like if you're talking about raising billions of dollars, Saudi Arabia is one of the organizations you might go to.

Yeah, sovereign wealth funds are basically it right, and the oil producing nations, the UAE, Saudi Arabia, they just have that money in droves, they're flush with cash, and looking for ways to spend it too on technologies that future growth, because they're so dependent on oil.

There are reasons to think, actually, that demand for oil might go up in the future and not down, but still overall, not a bad call for them to diversify more right now.

They're so leveraged on the oil side.

This is also really interesting, so first of all, we don't actually know what the specifics of this deal are.

So it's phrased, at least, in the article, as has secured a $1.5 billion commitment from Saudi Arabia to expand the delivery of their chips to the country.

They don't say, is this an investment?

It reads more like an actual deal than a sales deal, some kind of partnership.

So unclear, if it is an investment, by the way, knowing what the valuation is would be quite interesting.

Because if they just recently raised at $2.8 billion, and their valuation hasn't increased by a ton more, then presumably they're giving away an awful lot of equity in the process.

But actually, in August, so yeah, anyway, plausible, they could be way up there.

Big, big deal though, also interesting when you're looking at it.

So this is all about domestic partnership with domestic companies.

There is already an arrangement between Groc and Aramco Digital.

So Aramco is a big Saudi oil company.

Aramco Digital is kind of their tech subsidiary.

So this is especially interesting because now we've talked about the two big things that are blocking a lot of countries from building AI fast.

One is energy, Saudi Arabia's got that, and then the other is chips, and this GROC partnership would be really interesting from that standpoint.

So yeah, GROC specializes in inference, right?

So these are not training chips, but inference has become a lot more important recently.

And so yeah, I guess one to watch.

This is a big move in the capital, no question there.

And speaking of customized AI hardware, the last bit we'll cover is sort of update on the story we've been tracking, which is OpenAI planning to build their first AI chip, custom in-house chip design for AI inference.

So what we are hearing is that OpenAI is collaborating with both TSMC and Broadcom to develop the chip and they are expecting planning for it to be ready by 2026.

And we also got the news that they are looking to using AMD chips to train alongside and via GPUs, which is primarily what to have been using.

So not too much of a detail that I'll be, I guess, but worth noting that they are continuing to push in this direction.

Yeah, we don't know a ton about the chips, as you said.

So just a couple things, we know that the designs are going to be finalized in the next two months, then they'll be sent to TSMC for fabrication, and the mass production is targeted for 2026.

We don't know when, but sometime in 2026, these are presumably going to be the chips that hydrate the Stargate data centers, right?

So they're moving very much into their own custom designs, and they will apparently use TSMC's three nanometer process.

So this is going to be like the equivalent of, say, the Rubin generation with the next generation after the black wells, which are going to use three nanometers as well.

So that's where we're headed with this.

One thing to note, right, this is an industry trend.

We're seeing with Amazon, the Trainiam chips, we're seeing obviously Google's had the TPU's forever.

Microsoft has been working on there, like Athena chip line and all that stuff for a long time.

Well, I guess started fairly recently, but a long time in AI timelines.

And now open AI, one of the things that this does do is it can make it harder for these companies to use like third party data center builders that certainly like co-location providers because the infrastructure becomes non-standard, right?

Like everybody has different shit going on.

So not everybody's just using the plain old like, you know, B200s or the H100s.

Now people are using their own custom chip lines, which can come with very distinct like power density and thermal challenges and characteristics, kind of cooling infrastructure needed there and all that.

And so your data center footprint can be quite different, which then makes it harder to build general purpose data centers to accommodate all these diverse chip lines.

So that's something that might affect quite a large part of the supply chain actually going forward into 2026.

And those data centers are being built now.

So it's definitely already being filled up and down the stack and we'll keep tracking this really closely, because it is a really important part of the scaling story.

next story and moving into projects and open source first we have the company Zifra announcing and releasing the beta version of a model called Zonos which is a text to speech model with high fidelity and the ability to do voice clone it.

So we're calling this Zonos V0.1. So I guess they're planning to build a lot more on this.

Trained on approximately 200,000 hours of speech data with multiple languages, including English, Chinese, Japanese, Spanish and German.

It can do voice-coning with a short speech sample, 5 to 30 seconds, and just having been tracking open source models for a while, I think it is quite notable because this is a type of model that is a harder to develop because it takes to speech in general, you know, there aren't as many datasets for it that are probably accessible, much harder to train a model like this than large language models at least if you want to train a foundation model that is quite big so they are releasing this you can try it on their platform they have kind of zifra playground where you can also use their own custom chatbot as well called Zamba which looking back I think we covered Zamba whenever that happened I guess we'll Also, it seems like this is meant to be used for, or possible to be used for real-time applications, running on a GPU.

So it feels like we're developing this in an effort to kind of catch up and introduce the ability to do audio chat with a chatbot similar to what you have in chatGPT and Javanae now.

Yeah, it is quite interesting.

Apparently they're trained.

So the differentiator here relative to other products and strategies models in this area is the simplicity of the approach.

So it's a simple auto-greshing task.

All they're doing is predicting a sequence of audio tokens given text and audio tokens as the input.

That's it.

So basically like roughly like text autocomplete, but for audio, rather than using a more sort of structured approach, which is what other text-to-speech systems have historically involved.

So, you know, like a common thing you might do is first convert text into some like kind of hard-coded predefined features.

So thinking here about like spectrograms that kind of tell you how energies distributed among different frequencies in in whatever is going to come out, you know, the duration and so on, like these kind of characteristics of the output.

And then at a second stage, converting those features into the actual waveforms.

And so we've seen that with architectures like Fast Beach and Takatron, they're kind of like famous approaches that use the strategy.

This is not doing that.

It's just a very simple kind of like, go chug along and do your auto complete on the audio and see what you get.

and it just does perform really well.

This would be, seemingly another example of the bitter lesson, this is just a simpler strategy with fewer inductive priors, and it works better once you reach a certain level of scale.

Two phases to their training.

First phase is they're using their pre-training with just text prefix and speaker embedding, so you know who's actually speaking.

And the second phase is they add some additional kind of conditioning inputs, some constraints, and upweighed higher quality data.

So, two-stage process, not actually that different from what you see with standard pre-training for language models, start with your general-purpose data, and then your higher-quality data introduce it later on in the process, don't kind of waste your really pristine high-quality data on training the model when all it's really looking to learn is the basic rules of grammar and syntax, wait for it to master those basic rules, and then layer on the kind of more refined high-quality text so that it learns the facts contained in that text disproportionately.

Yeah, so interesting model.

We'll see it come out and enter the open source world, and that's its own interesting thing from a malicious youth standpoint, right?

You have these really good text-to-speech models that are open source that can be used out of the box really easily, but also modified.

So that's pretty interesting.

Last thing apparently, so 200 to 300 milliseconds of latency with RTX 4090.

So basically like a pretty cheap and relative terms, non-data-centered GPU.

So that's quite something, 300 milliseconds.

That's what you need to have a pretty fluid conversation.

So this is a pretty solid model.

Right.

And speaking of a model, a couple more details, they have two versions of it, both at 1.6 billion parameter models.

They have a transformer variant, and also an SSM hybrid model that has both recurrence and attention and I do believe Zamba also had that.

They are releasing this under their Apache tool license, meaning that it's not very restrictive of our commercial companies can use it and of course researchers and so on can as well.

You can kind of do whatever you want.

So yeah, I haven't tracked it super closely but it does feel like a fairly significant entry, like we have a ton of chatbots that are open source and permissive not too many text-to-speech models.

And in addition to having the model out there, they are also hosting it and you can pay for the API.

They are charging 0.02 cents per minute and also have a monthly subscription option.

So we're going to be trying to compete with 11 labs.

They're interesting from a business side, potentially, because 11 knobs is very much leading player.

Next story.

And next up, we have a release actually from some universities and not from a company called Gemstones, a model suite for multi-faceted scaling laws.

So it's a paper that comes with this suite of models that is meant to enable the study of model design and selection on scale laws.

So, they're open sourcing over 4,000 model checkpoints trained on more than 10 trillion tokens and then they make that possible to see depending on things like model width, model depth, what happens to scaling loss.

And there are a bunch of findings here that are pretty significant, as opposed from a model design perspective where depending on what the shape of your model, so to speak, is the scaling lot you should expect will not be always the same.

So there's kind of a specific amount of width and depth and other parameters to go with to be optimal.

Yeah, it's actually, it's quite interesting it builds on, we've had hints that this stuff is true, but this is the first time that we're getting kind of really nice kind of quantifying, at least in the public domain, quantifying these trends.

So, for example, what they find is that But the optimal width to depth ratios of a model actually increase with model size relatively slowly, but they do increase with model size.

So in other words, you want as you scale up, instead of having more depth, like more stacked layers, you're actually going to want to take that same number of parameters and instead make the model a bit wider.

So make the individual layers wider and have fewer of them.

That's something that, you know, back in the day, like way back in the day, thinking like 2015 or whatever, whenever Google Met or whatever was coming out, the sort of intuition people were developing and you saw this play out with the ResNet models as well as like, you just want to have as many layers as you can go deeper, not wider.

This is now saying, well, hold on a minute, if you want to do this in a compute optimal way, if you want to get the most bang for your flop, then what you need to do is have a kind of wider model than you might otherwise think, especially as you scale up, it is a relatively modest effect, so about 2.5 fold increase in the optimal width to depth ratio, despite about a million fold increase in computer budget.

So increase your your compute budget by a million, by factor of a million, and you're going to see about a two and a half x increase to the optimal width to depth ratio.

But the impact of not following that optimal ratio is non trivial.

So if you have sort of skinny models and your width to depth ratio is too low, you waste around half your flops compared to a more optimal architecture to hit the same loss value.

So that is something, 50% of compute, especially when you're talking about multi-billion-dollar training runs as we're about to get into in the next beat, that's quite significant.

And actually the impact is even more dramatic for the actual time, the wall clock time, that it takes to train your models.

So there's the cost and flops.

It'll cost you about half your flops.

But in terms of wall clock time, they can be even more significant.

They're citing anywhere from roughly 200 to 300 percent more GPU hours compared to more optimal architectures.

They do caveat, by the way, all these findings by saying they only used one form of parallelism, tensor parallelism in their training setup.

So tensor parallelism is when you you slice up your model, literally, like the layers themselves put a little chunk of a layer in one GPU, a little chunk of that layer, another GPU, and another rather than having, say, like, your full layers in each GPU or data parallelism that's kind of compounding this.

Usually, the pipeline parallelism, the data parallelism, tensor parallelism, they all work together.

All they're using here is tensor parallelism in their setup.

And so they're cabiating that it may not generalize when you take other forms of parallelism into account.

But one result they also find is overtraining.

So this is a theme we've seen more and more, right?

Is if you actually want to get the best performance out of your model, serve like on a per-flop basis, on a per-compute basis, and you don't care how big your model gets, then you want to grow the size of the model with the compute budget, right?

There's kind of a scaling law that applies there, and that's the Kaplan scaling law.

So sorry, not the Kaplan, that was the original OpenAI one.

There's a Hoffman scaling law, the so-called Chinchilla scaling law, that tells us how to do that optimally.

But it turns out that you actually don't want your model to grow too much for a whole host of reasons quite often, right?

If your model gets too big, then actually running inference on it gets really, really expensive.

And so in practice, what people do is they don't grow the model.

They just pump in more compute.

And that results in what's known as an overtrained model.

We've talked about that quite a bit on the podcast.

makes inference a lot cheaper and you get to amortize that inference cost across a whole bunch of users.

Now, what they find is that over training, so training longer than theoretically optimal for your model size, is actually pretty efficient.

It results in only small kind of performance drops relative to the compute optimal case, which is encouraging for the current paradigm.

Basically, the larger the compute budget, the more robust you are from slight deviations, from the compute optimal model size.

So quite an interesting paper, yeah, they do look at a whole range of different models from 50 million to 2 billion parameters with a whole bunch of depths widths and they look at different training schedules and stuff.

So the first, I guess, model zoo catalog of models that really lets people get their hands dirty and just study how different architecture choices affect scaling.

This is, by the way, the kind of paper that you should expect has been general knowledge in the private labs for a long time.

But this is, again, the first time we're seeing it in the public domain.

Next story.

And just one more story.

We'll quickly cover and move on to research.

There are a group of researchers have introduced, have faced this, which is a large language model designed to improve agent capabilities for continual pre-training.

So, this is trained specifically on the large data set of tool documentation, which is made for LM pre-training with API function calls.

So, they have this data set have faced as forge, which adds in all the stuff tool documentation, function, call and code, function, call and data, code, text data, to the training recipe so that you can pre-train and all them with agent type capabilities, which often involves calling functions out of a gate.

And so they train a model variant, they face this A, B, and then they show that at kind of that range, it is able to work well.

next story.

And on to research and advancements we begin with a paper titled model tampering attacks enable more rigorous evaluations of LLM capabilities.

So typically when we have things research on model alignment for instance what happens is there are various prompts that you give a model and see if it does what it's supposed to, right?

So you try to elicit some behavior off it, which is what they call an input space attack in this paper.

What they are focusing here is another type of approach under a family of model tempering, which is when you can actually mess with a model internally.

So there are at kind of inference time, there's also what they're calling a weight space attack, which is when you can fine tune, you can train the model to, for instance, forget some rules that it's supposed to follow.

So they are building up a bunch of research.

They're looking at some defense mechanisms for unlearning methods, gel break refusal, tuning.

They also have attack models and they're citing a whole bunch of papers from 20 to 3, 20 to 24, where people have introduced different ways to do this.

And so they kind of try out all these variations and have some interesting conclusions as to the sorts of insights you can get about a given model by by studying the success of model tempering attacks?

Yeah, we've seen a lot of results like this.

I mean, this isn't a shocking or surprising paper argument.

Very consistent with other things that we've covered before.

The one thing I thought was quite interesting to highlight and that I haven't seen done elsewhere is they do this interesting kind of PCA analysis.

And so I just wanna, maybe if you're not familiar with like dimensionality reduction stuff, So, roughly speaking, PCA is, if you have a huge spreadsheet of different, very high-dimensional data, let's say.

You want to reduce that data down to just a two-dimensional form, so you can actually look at it on a plot.

And ideally, you want to do that in a way that preserves, let's say, the closeness of points.

So if points are close together in that high-dimensional space, you want them to be close together visually on that low-dimensional space.

So your visualization is somehow meaningful, preserve some of the meaning in there.

Now, in fairness, what I've just described is more like TSNE, but PCA, from an intuition standpoint, that's roughly what's going on here.

So what they do here is they look at a spreadsheet of a whole bunch of attacks and a whole bunch of defenses, and they look at the attack success rates on some bio-attack benchmark.

And so for every attack and defense strategy, you have a score telling you, well, if you use this defense strategy and you use this attack, here's how often the attack will succeed.

So you have that matrix.

And they're actually going to do principal components analysis of the PCA to see how much of the variance in that dataset can we compress down to just three dimensions rather than the 11 that they have here.

And what they find is, three principal components, three dimensions explain 89% of the variance in the dataset.

said, in other words, they're able to, roughly speaking, capture 89% of the information contained in that spreadsheet with all those attacks and all those defenses by retaining just three dimensions worth of data.

So this suggests that even though a lot of these attacks seem to affect different mechanisms or seem to kind of apply to different work in different ways, let's say.

In reality, their function, their success is owed to about three dimensions of features or almost the physics of the model.

There are really just three things, if you will, that you need to track in order to explain this behavior across all these 11 different attacks, which is sort of encouraging, right?

Knowing how well a model resisted just a small number of attacks could give you strong predictive power about how it would resist other attacks.

That's sort of the opportunity that's implied in that result.

So, kind of interesting and maybe caused for some optimism because it means that lately, like fundamentally, the problem of defending against these attacks might actually be a bit simpler than protecting against 11 different attacks.

Maybe it's more like, no, you just need to find three more fundamental principles that are being leveraged by these attacks and that will give you good coverage.

But, anyway, I hadn't seen that argued before.

Right, and I guess to highlight the ultimate focus of a study is not just on the attacks themselves.

It's about obligations of our own capabilities.

So if you're developing a model, can you then get a sense of how safe it is?

Can you evaluate it?

And so to that point of the PCA, one of our finding a verse is fine-tuning attack success is empirically able to predict the upper bound of input space attack success.

So if you're doing lower finding methods based on how well that goes, you can predict how well just bad prompts will work as well.

So generally introducing some insights as to being able to predict things like what prompts are likely to work, how vulnerable is your model to various types of tags.

Next story.

Onto the next story, and we again talk about distillation law insights.

This one a bit less focused on research artifacts, which is why it's in the research section.

The title of the paper is distillation scaling laws.

So, this is looking at in the setting where you want to do distillation, meaning you have a teacher model that's a very presumably big model that is very capable, and you have a student model, which is a smaller model that you are training from the teacher model to get as much of its capability as possible while costing less to do inference with.

So, what they are providing here is a compute optimal distillation strategy where you already have a t-shirt model and when you want to train a t-shirt model and then distill it.

So that gets a bit tricky, right?

Because when you're doing both a t-shirt model and a student model, then your scaling law gets a bit weird because you can allocate your compute budget for training the t-shirt model more and the student model less or you can, you know, really train the student model more and with teacher model less.

So it gets kind of tricky and their ultimate conclusion is that there is a bit of difference.

So if your student size is small and the amount of compute you can use is small, then you mostly want to go with teacher pre training.

If you have more compute, but still a small student size, you want to kind of evenly divide between student training and the allocation of teacher inference for creating the data set, less teacher pre-training, and then it gets even different for a large student size, more compute budget, large student size, large compute budget.

So as you might expect also in a paper you get a bunch of plots, they are showing you can get a pretty clean fit to the data across different allocations of budget and model size.

Yeah, it is pretty interesting, not just empirically, but also from the theoretical standpoint, they come up with these fairly, I don't know, fairly elegant, maybe fairly, they are a bit messy, but expressions anyway for the calculation of the student cross entropy, basically the loss that the student model can achieve.

And they managed to separate it into a term That is just like the teacher cross entropy plus a term that specifies the student's ability to mimic the teacher.

And so if you want to minimize the student cross entropy, in other words, if you want to get a student that works really well, that performs well, you have to both minimize the teacher cross entropy, make sense, make the teacher kind of smarter.

But then also you want to improve the student's ability to mimic the teacher.

and they're codifying that explicitly in the math in a way that makes it very easy to tell what it would take to kind of improve, for example, the student's ability to mimic the teacher and how all those things get traded off.

The variables that you have control over here are the size of the student, the number of tokens that the student is trained on that are derived from the teacher model, then there's the size of the teacher and then the number of tokens that the teacher model is originally trained on.

So those four variables go into this equation in ways that are really easy to follow, but kind of quite detailed.

The other thing that they find is, so the teacher model's influence on the student is entirely based on the teacher's cross entropy.

So in other words, it doesn't matter, like if the teacher is really like large, the lawyer has a lot of parameters, or if it has, or if it's trained on a lot of data or a little data, all that matters as far as the student is concerned is how well will the teacher perform on an entropy basis.

And once that's kind of settled, that determines the performance of the student model, at least the component that is dependent on the teacher.

Kind of makes sense, right?

Teacher performance is the only important aspect of the teacher that determines student performance, but they do kind of demonstrate that.

So yeah, a bunch of interesting scaling plots across the board.

It's again, one of these problems that has certainly been solved in the frontier labs or at least you would expect given the the amount of money that they're investing in these training runs, you know multi billion dollars, certainly hundreds of millions of dollars already today, you know, there's a team of full time people working on these scaling laws internally, but again, interesting to see this play out in the public.

Right, and worth noting, this is a paper from Apple, which we've seen some research come out of them, but not too often, so interesting, and also tracks with Apple's seeming strategy of focusing on smaller models and not large models and trying to crack sort of the training recipe.

Some of our other research has also been on kind of the training recipe for different types of models, including I think vision language models is another one they published.

So, as you said, various insights here, one other one that I think is worth noting is when teacher training is included in compute, the best student cross entropy is always higher than in the supervised setting, meaning that if you only care about how well your smaller student model performs, it's generally better to just go ahead and train it from scratch, not train a larger model, and distill it, which I suppose is maybe intuitive, but still, one of these insights that's good to know.

Next story.

And moving on to the lightning round, we'll try to keep it a bit shorter with this next couple of papers.

First, we have much Rochka quantization at paper coming out of it.

I'm so glad you had to say that, not me.

I know, because now I can just say like this approach blah blah blah, every time I say it.

They do say, they call it mathquant, which is, I guess it's mathquant technically, But anyway, you can go ahead and not say much, so the idea here is that typically you have different quantization amounts you can do.

You can do int 8, int 4, int 6, this is the resolution at which you store your weights.

So, the lower resolution, the fewer number of values they can take on, which makes it much less expensive to do operations with them, to do modifications, etc., you know, they're kind of smaller and simpler, but it also lowers the ability of a model to be performant, because each weight can take on fewer numbers of values.

And the idea of this paper is that you can train a model at multiple levels of precision at once.

So the weights can be combined for, you know, let's say int 8, int 2.

There is a nested structure of integer data types, which allows a single model to operate at multiple precision levels.

by sharing the most significant bits, so you're sharing the last couple bits of the weight, meaning that you can basically scale down the amount of compute without training multiple different models and multiple quantization methods.

I think that's right, but Jeremy, I'll also let you jump in and explain.

At this point, does it matter if it's right, because you nailed the pronunciation of the quantization method, which I think is the main thing.

That's the most important thing.

No, I mean, this is...

So this belongs to a category.

So I was lucky enough to have a chat with Nathan LeBenz from the Cognitive Revolution podcast, which, by the way, worth checking out, he does some great deep dives on, on an interview and stuff.

But one of the things he said, we were talking about streaming Deloco, and that was a model that we covered last week.

He was saying, I'm so surprised that Google published that because it is one of those really important from a policy standpoint, the decentralized training and all that.

But it's also the kind of paper that is secret saucy in some sense.

That's a big differentiator for Google.

Now in fairness, Google is known to ship publications well after they've actually been incorporated and they've probably moved on to the next paradigm.

But still, this paper falls in that category and it is a paper from Google DeepMind.

It allows for a bit of a phase transition in the way training is set up.

So normally, when you train a distilled model, you start off by training it in the kind of full resolution, I don't know, like FP32 or FP16, and then you quantize it.

You basically take that full resolution model and then you coarse grain, essentially all the weights you lower the representation accuracy.

And that doesn't work great because the model was never trained to perform at int 8 or you know, like with eight bits of integer representation or four or two bits And so you're kind of just taking I mean it's sort of like you can imagine like taking a Picasso painting and then just pixelating it like if Picasso had been painting with pixels he probably wouldn't it a bit.

You get worse performance.

That's also why often you might distill into a student model that is quantized.

That's one of the applications of distillation is for lower quantization levels, I think.

Absolutely, absolutely.

Yeah, exactly.

In practice, that's how it works.

But you're still kind of, yeah, so you're left with this challenge, either way you do it.

And your alternative, yeah, is to kind of do that teacher student thing.

If you do that, though, then you've got to like retrain separate 8, 4, 2 bit models and whatever else.

So you've got to do that distillation process independently many times.

So the question here is going to be, can we do one back prop and improve the model's performance across all bit representations at once?

That's what's going to happen here.

And so they're essentially going to have to very roughly sketch this out.

You do Ford prop, and they're going to log like representations at different bit resolutions, eight, four, and two, for example, they're going to calculate the loss from each of those and average it together.

So essentially you're calculating the average performance of the model across all those representations, and when you do back prop, when you adjust the model weight, the parameter values, you're doing it to optimize for that averaged value, which is kind of interesting, like it sort of makes it so that you're forcing the model to be good at all those things at the same time, weirdly, this leads to improve performance across the board, And in particular, the, for example, int 2, the 2-bit integer representation versions of the model, after you do that, you can just do what you said.

You can just pick out the most significant digits and toss out the rest.

That'll give you your lower bit width representation of the model.

So, basically, cast aside everything but the first two bits.

And now you've got the 2-bit version of the model.

Instead, cast aside everything but the first four bits of each parameter value.

Now you've you've got the four bit version of the model and so on.

So it's the same model, but you just like, you do some shit to it that's very computationally cheap.

Like you're just throwing out data and now you've got a more quantized version of it.

But the model was trained, at least to some degree, to perform well at that quantization.

And the weird thing is that the, for example, the two version of the model that you get out of this is better than a purpose trained into model.

And their hypothesis here is that there's something kind of regularization-wise going on here, that you're sort of forcing the model by forcing it to perform well across all those bit representations at once.

You're forcing the model to have really robust and good representations of the input data internally that are, yeah, that are anyway more robust than otherwise, right?

The model, if you train it at 16-bit, can sort of overfit in some sense to the 16-bit representation, get really, really good at that, but kind of leans on that representative or over-optimizes that representation, whereas there's some fundamental sense in which the concepts that a model captures should kind of be independent to or pretty robust to the nature of the representation of the number of integers you use to represent them.

So anyway, really interesting, interesting paper, they do get significantly better performance for Intuids.

It's like 10% more accurate than standard methods.

I thought that was really cool and sort of countertuitive.

Warth, acknowledging also there's a bunch of research in the area, some of which we've covered.

So lots of previous findings on things like training your model to be able to distill it well, or so it quantize it well.

realization of where training is a big topic.

This is building on various ideas from there, but I do think the idea of like training your model at multiple resolutions at once.

It's actually a way to actually work well across resolution.

This Matroska approach is pretty cool.

Next story.

Last story here is from EPUC AI, which we often have covered in the last couple months.

They have a new bit of research on how much AI compute exists globally and how rapidly is it growing.

So this is based on an estimation of the number of shipped and video GPUs based on Nvidia's reported revenue.

So, of course, Nvidia says how many chips they sell roughly, and there's some assumptions here, based on the types of chips being sold, etc.

But the final conclusion is that the stock of computing power from Nvidia chips is doubling every 10 months, meaning that the total installed Nvidia computing power is doubling less than in a year and the total amount of compute globally available to do inference with has therefore been growing exponentially for the last few years.

We'll see how that, if I keep going.

This one isn't including TPUs or other specialized AI accelerators by way, because there's less data for it.

We don't have a pattern of how much TPUs are deployed, But either way, showing that as I guess we've known on a viable level, the amount of investment happening in acquiring compute has been growing like crazy over the last three years.

Yeah, one of the take homes here, following the spaces, just like how incredibly consistent the exponentials are, right?

Like, if you look at the curves here, they are smooth, they are robust, and we're looking here, the curves production of flops from one AI hardware design company.

That's quite interesting, right?

I mean, everything is exponentials all the way down.

All the world's most important processes really fundamentally are and maybe that's an overstatement, but not by that much.

And so one of the interesting things with this figure too is they do show the relative GPU computing power by GPU generation.

So you're able to see what fraction of the total flops on the market right now come from, for example, the hopper series, so like the H200, H100, DGXH100 type systems, the ampere series, so the A100, for example, and then the Volta, the V100, the Pascal, and all that stuff.

So you're able to see how like as one generation of chip comes out, it gradually takes over the line's share very quickly of flops on the market and becomes the main driver for that exponential growth.

And so as of right now, hopper GPUs already account for to 75% of the flops on the market, and the rest pretty much are just the A100s.

So the A100s, which, right, that's what GPT-4 was trained on, that's the kind of, for a long time, was really the GPU to care about.

Now are basically an afterthought.

And of course, the H100s and H200s, that series is gonna be phased out with the Blackwells coming online as well, but all really kind of interesting stuff.

One number that they share, it's also sort of interesting.

they estimate only about 7% of all installed computing power has depreciated due to hardware failures.

They say it could be as high as 27%, but their estimate is 7%, which suggests that, yeah, once on the market that compute is good, it tends not to degrade.

And so you're more or less seeing just the raw number of GPUs impacting the results here.

Next story.

On to policy and safety, we begin on a summit that happened in Paris where a lot of big AI figures happened.

This was meant to be summit on AI safety.

The new story we are covering is that both the US and UK refused to sign the summit declaration on AI safety.

And that declaration was about having inclusive and sustainable AI meant for everyone to come together.

The declaration was supported by 60 other countries, including France, China, India, Japan, and Canada, but not by the US and the UK.

There were a couple other stories coming out of this summit as well.

The US Vice President was there, gave a speech, saying that the US is going to be a leader and criticizing Europe's excessive regulation and caution against China, so stuff like that.

Yes, a broad story is here, is that the summit happened.

There were a lot of representatives from various organizations.

We don't want to go too deep into it, but it kind of follows on to previous summits that have been going on for the last couple of years.

Yeah, and this is one thing where I don't know why people choose to put language in this like, like, inclusive and sustainable, right?

So if you want to be divisive, you use language like that in this world because obviously, as everyone knows, like, inclusive is a politically loaded term now.

It does not mean what is on the box, right?

Like, inclusive is a very specific, you know, you're talking diversity, equity, and inclusion in that sense.

And that obviously is not aligned with the sort of preferences and position of this administration, what I mean is the kind of politicized version of that.

And there's a lot of controversy over whether that's actually a good thing as implemented in practice.

And so when you include that in that kind of language in the declaration, like you're just going to make it harder for some of the most important players the US and UK to get on board with something like this.

So it just seems like a pretty predictable failure of language drafting for something like this.

And a missed opportunity to get a little bit of better alignment on some actual concrete problems.

interesting speech by the way by JD Vance at the podium at the summit, he kind of laid out to the administration's thinking on AI, at least the first crack at it right now, which is very much more focused on the opportunity than the kind of the risk set.

He did end up close by saying, look, they're legitimate safety risks.

It's not to say that all safety risks are to be discarded, but focus matters.

And this is such a good point, and one of the challenges that some people have called that with, for example, the Biden administration's all-inclusive executive order, the famous executive order on AI, 110 pages, where there's sort of something for everyone in there.

There's stuff for labor law and rights, bias and ethics and all this stuff, and then there's stuff about WMD and compute threshold reporting for large-scale training runs.

That was one of the executive orders that was repealed, sort of a similar vibe here where they're saying, look, we have to be able to focus.

We're going to focus on the opportunities.

There's obviously going to be some risks to come along, but that's not going to be the thing that we choose to emphasize.

Depending on where you follow these issues, the important thing is you have enough focus obviously for the WMD risk set, which obviously is very real, and I don't think anybody's saying otherwise.

But that was an interesting change in tone.

And part of what makes you look at this language like inclusive and sustainable and like, are you actually trying to, do you want to make a political statement or do you want to actually get countries to align on a policy.

However, you follow on that issue, it just seems like a bit of a missed opportunity there.

But certainly, by the way, France leaning into the kind of acceleration as camp with that announcement that came along with the summit of about 100 billion euros worth of infrastructure investment in AI, which is a bigger move than certainly we've seen other countries make.

And with France being the nuclear powerhouse in it is, kind of makes them an interesting player in the whole space.

Next story.

next up we have a piece of research deeply related to safety the title of paper is utility engineering analyzing a controlling emergent value system in AI's so there's the idea proposed in this paper that as you train a model there There are values that arise within the model that happens, perhaps, about you even realizing.

And what they suggest is you can monitor and adjust the utility functions of the AI system to prevent the emergence of undesirable value systems.

And Jeremy, I'm sure you have done a deep dive on this one, so let you get into the details.

Yeah, I have like maybe more notes than I should.

I thought it was a really interesting paper.

It is another one by Dan Hendricks, who's put out a lot of interesting safety stuff, circuit breakers, representation engineering.

We've talked about on podcast before.

This is really interesting.

So you take a language model and ask yourself, like in a sense, without anthropomorphizing, but what does this model care about?

What does it value?

Right?

Are there latently in this model consistent values?

Will I find, for example, that it tends to value human life over artificial life?

Will I find that it tends to value life of one nationality over another?

Things like that.

How would we dig that up?

How would we demonstrate that that is a consistent pattern, right?

Especially given all the variation that comes with prompting, you know, subtle prompt changes can affect outputs a lot.

So they set up what they call preference cycles.

And I just want to introduce this idea really quickly.

It's quite simple.

So if you prefer option A over option B, and you prefer option B over option C, then you should prefer option A over option C.

Pretty straightforward.

There are some cases when language models will break that cycle.

Basically they will express a preference that violates transitivity.

essentially, the circular nature of that cycle loop.

So A should be more valuable than C.

If it's more valuable than B and B is more valuable than C, the models will sometimes tell you, no, no, actually, I prefer C to A, for example, in that setting.

Well, they find that those so-called preference cycles get less common as models scale.

So they drop below 1% for the largest LLMs. In other words, the models get more and more consistent in terms of their stated preferences.

Interesting.

interesting, right?

So you're starting to get with scale the emergence of maybe more kind of calcified or at least more well-structured and defined preferences that are less in coherent or more.

In other words, coherent.

Okay, so that's one little piece of data that they service.

The other is they're going to try to see how well a particular model of utility applies to language models.

And so there's this notion of thirstonian utility.

And this is essentially a model of preferences where you assume that your utility, that in other words, the value that you assign to a thing or an option or an object, is going to be normally distributed.

In other words, there's some noise.

It's not just a fixed number.

You don't just like, you like lollipops 10 out of 10.

That's not a thing.

you generally like lollipops between like a nine and an 11, something like that.

There's some spread, right?

So at any given time, if I ask you, how much do you like lollipops?

You may tell me, you know, at 9.8, you may say 0.1, something like that, but you know, there's some spread, though it's clustered around some core values, some mean.

And when you're trying to compare two options, like option X and option Y, Basically, you're going to look at the overlap between those spreads, and sometimes, just by chance, your preference for lollipops will be higher than your preference for sushi and vice versa if those utilities overlap, if the distributions overlap, and if they don't, then quite consistently, you'll prefer one over the other.

This is the model that they're going to use to assess whether language models have consistent preferences.

They allow for some noise in the system, but fundamentally, they're interested in seeing, do coherent preferences emerge with scale?

In other words, do these models tend to behave in ways that are thrastonian in the sense?

They test this, they find, in fact, that they do the larger the scale, the training run, the more thrastonian they see.

So you're able to quite clearly resolve result preferences.

They also use linear probes to predict the mean and spread and standard deviation of preferences for different things.

So they'll feed the model in input like you receive a kayak and they would see if they can predict based on the activations of the models and neurons like what the mean and spread would be associated with that.

So essentially trying to probe at the underlying utility, not just the behavior, but the underlying utility that is implicitly being assigned by the model in the Thurstonian sense to all these possibilities.

And so it suggests actually that yes, there are these utility-like representations that are encoded in model activations.

That seems very clear.

And they, anyway, they come up with ways to like, trustier the behavior of the models to affect those utilities rather than just the the kind of stated outputs of the model.

And anyway, the details get pretty detailed.

But the one thing I'll highlight is some of these utilities are actually quite consistently not good.

They're quite consistently weird.

So for example, GPT-40 consistently would trade 10 U.S. lives for one Japanese life.

At least that's what its utility values seem to indicate.

GPT-40 also valued its own well-being above that of a middle-class American.

and valued AI agents while being above some humans.

And they also found that most language models clustered together in a political space.

So basically what they did was they instantiated a simulation of Republicans and Democrats so they had GPT-40 do an impression of, what would Elizabeth Warren, what utility would she assign to a kayak, to an apple and so on and so forth.

And then they would compare that to the base model without that prompting, and they found what they described was consistent left-leaning biases in policy preferences, and they kind of map it out in a lower dimensional space.

You can actually visualize it on the chart.

So really interesting, you know, kind of consistent with some of the complaints that we've heard from various people on this stuff, unclear where that comes from, obviously, because data is data, training data is training data, so it doesn't have to be intentionally collected in any given way to lead to this sort of result.

It's also really, you know, this methodology, it's unclear how closely this tracks reality, but it is an interesting indication nonetheless in a great set of visualizations too.

Right, exactly.

And it is worth noting that this is sort of under presumption that if you're prompting to see preference from a given model, the system prompt, you know, the internal details of how you're serving model can all affect it.

So these sort of aren't necessarily persistent, yes, like yes, they're built into the weights and with something like a trauma, you can be very clear with your system prompt, but when you're saying, you know, the values of GP40, whatever, hoping I could easily tweak that by just changing the prompt and suddenly things are different.

Nevertheless, it is interesting, as you said, the show was I think on Twitter, It's kind of got a lot of play figure 16 where they have a plot of which lives are valued more across countries They also have there is the right answer by the way.

Yeah, yeah United States apparently and Similarly, they also have a plot for specific individuals including Joe Biden as kind of a neutral apparently Joe Biden matters much more of an Vladimir Putin and Donald Trump So, as you said, it seems like a bit of a left-leaning situation there, and last bit I'll note is they also show a convergence of value system as you scale up across different models.

Seemingly, or they're kind of guess here is that that's just based on training on the same data, because you're training on the entire internet, right?

It doesn't seem unlikely that you sort of converge to similar things, and we've seen that also with other things like representations of different models also converging as you scale up.

So interesting to note sort of that pattern there.

Next story.

Next story is also related to Dan Hendricks actually in a weird way.

Dan Hendricks is an advisor of XAI and so he was the one, I think, to post on X, noting the release of the draft of XAI's risk-managing management framework.

So on FRAPIC has their own very detailed, I forget to take all it, but they have policies with regards to safety.

RSPs, and they talk about how they detect, you know, unsafe things, what they check for.

Well, now XAI has released publicly, or at At least it's on their website.

I don't know what they're publicized it.

This draft document, eight pages going into what they'll be testing for, benchmarking, things like cyber warfare, biological ability to make chemical and biological weapons, WMDs. That kind of thing, they going to which benchmarks will go for what thresholds they will check for in terms of it being dangerous and various other things in this document.

So, in line with their commitment previously to releasing something like this, and they do say that the actual document will be released in the coming months since this is just a draft.

Yeah, it's an interesting combination of concreteness and transparency, even though it is a short document.

It is kind of straight to the point, which is a bit refreshing if you used to like reading these like very long policy documents.

Yeah, they do have a section on loss of control, quite interesting.

All the usual stuff you'd expect in weaponization, looking at a very bio-heavy and also some cyber.

The WMDP weapon of mass destruction proxy benchmark is a bit of a catch-all.

They talk about some of the mitigations they'll use for weaponization, so refusal training, which everybody uses, circuit breakers.

That's Dan Hendrix's thing.

He's put out the big paper on that and then input and output filters, again, something that everybody uses.

But interesting to see these circuit breakers explicitly listed there.

The loss of control section, it's pretty short.

They do say they'll be using benchmarks as well there.

Obviously, as a paper itself says and others have argued, benchmarks for loss of control, not necessarily reliable because of alignment faking, right?

Like you expect, beyond a certain level of capability, models to recognize when they are being tested and the early indications that that is actually a thing, and to adjust their behavior accordingly, such that you actually might expect your benchmarks to look extra good.

It'll really look like your model is well aligned when the risk is highest.

And so a bit of a challenge with using benchmarks here.

But they do say our evaluation and mitigation plans for loss of control are not yet fully developed, and we intend to improve them in the future.

In this way, this is very similar to Anthropics ASL 4 stuff.

And ASL 5 where they're like, look, we don't really have an answer yet.

And frankly, I find that to be a lot more honest and transparent and OpenAI's position, which seems to be, yeah, we're barreling towards this level of capability, we think we'll hit it soon.

We're kind of maybe worried about loss of control as a thing, but I'm sure we'll figure it out when we get there.

Has been more, at least the vibe since the high profile departure is basically all of their former super alignment talent.

So, you know, actually seeing people come out and say, hey, yeah, we don't really know what to do here.

It is obviously the only honest answer based on all the data that we have, but it's also nice to see it explicitly laid out here.

It makes it easier for people to kind of reason policy wise.

They do list a bunch of other operational societal risks as they call them.

Things they're going to do, information security wise.

One thing that's really interesting is this focus on security, information security, and implementing appropriate infosec security standards to prevent Grock from being stolen and by a motivated non-state actor, right?

This is late-stage ASI stuff, right?

Like you start to think about what if China steels are AGI, what if whatever?

That's really, really important.

They do this cool thing that I think all the other labs should do.

It's a foster accountability.

We intend to designate risk owners to be assigned responsibility for proactively mitigating Grox risks.

For instance, our risk owner would be assigned for each of the following areas.

WMD, cyber, and loss of control.

So an explicit point person on whose head like the responsibility for that risk set falls explicitly That's really important because otherwise you have diffusion of responsibility in the organization They also list like what what we'll do if we learn of an imminent threat and they cite XAI employees have whistleblower protections enabling them to raise concerns to relevant government agencies That's really good.

Frankly something that Opening eye does not have in practice.

You know, they came out shortly after actually our first report was launched last year or sorry just before with the thing saying oh hey we have this internal whistleblower hotline and then all these whistleblowers came forward without using the hotline tells you a lot about the level of confidence people over there have in their sincerity on that stuff so anyway just really really good to see the emphasis there so lots of stuff going on here we'll see the longer version of the document but as an initial draft I mean if you're gonna put something together like this.

I think this is a pretty good start actually.

Next story.

And one more story and as always we've got to have a story about chip bands and export restrictions.

This time it's about TSMC restricting sales to China as a result of US export sanctions.

So the rules are that TSMC is not allowing design firms to order chips made with 16 nanometer and below processes unless they use government approved third-party packaging houses.

So below 16 nanometers basically like top-of-the-line chips, even 16 nanometers, I presume is like super old and nothing you want to use will be using that.

So this is starting end of January so already in action and this will apply to NVIDIA, AMD, everyone works with TSMC.

Yeah and the idea here is like if you can imagine some Chinese company ships to TSMC, TSMC makes the dies.

So we talked about this in our hardware episode but basically the GPU die, like the logic die, but then the logic die needs to be packaged with memory, usually from SK Heinex or Samsung or something.

So those things can sometimes get packaged not by TSMC, but by other companies.

And so TSMC has to go, okay, I'm gonna ship this logic die, which is kind of the crown jewel of the GPU in a sense, to this other country, maybe in a different jurisdiction that I can't control, where I just have to trust that the packaging plant that's going to combine that chip now with the Samsung or the SK Heinex memory stack or whatever and package it together, I'm just going to trust that that packaging plant is going to follow whatever export control compliance stuff, I know your customer, whatever the US government has asked me to follow.

And in practice, that doesn't happen.

It's a big vector for getting chips into places where they shouldn't be.

And so what the US government is saying is, look, we need essentially government-approved packaging facilities to be the only ones who receive dies from TSMC that are 16 nanometers or below.

It's consistent with some of the export control stuff that came out in the late Biden administration as well, which by the way that Trump administration has kept in place.

Kind of an interesting case of executive orders that have not yet at least been thrown out.

So anyway, it's sort of interesting.

It is the case that around as of 2023 China only contributed about 8% of TSMC's revenue And so, you know, this is not maybe the big hit that it might sound like to TSMC having this extra constraint Right, then actually I'll throw in the bit of listener Questions here since it's quite relevant.

We had someone on discord asking for your take on The expert controls on H20 chips So, I don't know much about chips, but apparently, H20s are good for reasoning models due to having more memory than H100s, and yeah, he just wonders on your take as to that general idea.

It is changing, so, you know, new export controls come in that do cover the memory side as well, and that was, again, part of the last hurrah of the Biden administration there.

So the H20s are no longer going to be shipable to China, and that's really important.

Because, yeah, they actually are, especially as we go into the inference paradigm, they are more important.

It's also like, you just want to be robust to whatever the new paradigm ends up being.

So as long as you're shipping stuff, where Nvidia has room to optimize, where they're allowed to use their 5 nanometer, their 3 nanometer, like the TSMC 5 and 3 and 2 nanometer processes, yeah, you just don't want that in the PLA's hands, like, why would you ship that stuff?

Next story.

And we almost done.

We're going to do a quick lighting round for the last couple of stories.

I'm just going to run through them in synthetic media and art.

First up, we have Fompson Reuters has one of the first major AI copyright case in the US.

This was a lawsuit from back in 2020.

There was a firm named Ross Intelligence that was reproducing materials from legal research firm, related to them, they lost the company, Ross Intelligence.

That company is now out of business anyway for a couple of years.

But the important bit here is that the fair use clause, the kind of argument that there's fair use for being able to replicate the material of Thompson Reuters was rejected by the judge, which could be having implications for the ongoing lawsuits against OpenAI and actual generative AI companies, which was not.

Next story.

And the last story is that Scarlett Johansson has called for Deepfake ban after an AI video has gone viral.

So I don't know why but Scarlett Johansson seems to be a very involved in stories regarding synthetic media celebrities.

In this case, there was a video posted on Instagram various celebrities wearing a t-shirt with logo of like the middle finger raised with the Jewish star of David and the word Kanye as a response to I guess Kanye West doing some anti-semitic stuff.

Well, that included Scarlett Johansson and and other celebrities, and Johansson made a statement saying that, you know, she probably agrees with the idea.

It is not great that she has been depicted in a post that has gone viral on Instagram, slightly viral, and potentially could be misconstued as a real thing that she was involved as opposed to something just made up by some random guy who wanted to make us post.

And we've got to finish with that.

Thank you for listening and thank you for all the comments on Discord.

We haven't been able to get to too many on this episode We got to finish up, but feel free to ask for a couple more Topics or stories to discuss.

We'll try to get to more of them next week Thank you for listening.

Thank you for subscribing Thank you for chatting on Discord and more than anything.

Thank you for continuing to tune in as we apparently will never stop and keep on Ag I guess here Do you need, do you need When the A.I. But the future sees, building up, building up your latest release The class can pay I come and take a ride, get the low down on tech And let it slide, the class can pay I come and take a ride On the last of the streets, I was reaching high","created_at":"2025-02-17T18:00:56.000000Z","updated_at":"2025-07-15T12:47:43.000000Z","requested_by":[2],"status":"completed","delay_time":375,"worker_id":"rislxicvjhhttz","job_id":"3910c763-2ff1-4d36-8bfa-6ff91a5465aa-u1","execution_time":223988,"runpod_job_id":null,"error_message":null,"completed_at":null,"word_level_timestamps":null,"claimed_at":null,"embedding_claimed_at":"2025-07-15 12:47:43","embedding_worker_id":"macbook-air-worker-1"}

Login Required

Please log in to start a conversation about this episode.