The a16z Show - Tesla's Road Ahead: The Bitter Lesson in Robotics

Tesla's Road Ahead: The Bitter Lesson in Robotics

Oct 24, 2024 - 36m 46s

What does Rich Sutton’s "Bitter Lesson" reveal about the decisions Tesla is making in its pursuit of autonomy?

In this episode, we dive into Tesla’s recent "We, Robot" event, where they unveiled bold plans for the unsupervised full-self-driving Cybercab, Robovan, and Optimus—their humanoid robot, which Elon Musk predicts could become “the biggest product ever.”

Joined by a16z partners Anjney Midha and Erin Price-Wright, we explore how these announcements reflect the evolving intersection of hardware and software. We’ll unpack the layers of the autonomy stack, the sources of data powering it, and the challenges involved in making these technologies a reality.

Anjney, with his experience in computer vision and multiplayer tech at Ubiquity6, and Erin, an AI expert focused on the physical world, share their unique perspectives on how these advancements could extend far beyond the consumer market.

For more insights, check out Erin’s articles linked below.

Resources:

Find Anj on Twitter: https://x.com/anjneymidha

Find Erin on Twitter: https://x.com/espricewright

Read Erin’s article ‘A Software-Driven Autonomy Stack Is Taking Shape’: https://a16z.com/a-software-driven-autonomy-stack-is-taking-shape/

AI for the Physical World: https://a16z.com/ai-for-the-physical-world/

Stay Updated:

Let us know what you think: https://ratethispodcast.com/a16z

Find a16z on Twitter: https://twitter.com/a16z

Find a16z on LinkedIn: https://www.linkedin.com/company/a16z

Subscribe on your favorite podcast app: https://a16z.simplecast.com/

Follow our host: https://twitter.com/stephsmithio

Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.

Stay Updated:

Find a16z on YouTube: YouTube

Find a16z on X

Find a16z on LinkedIn

Listen to the a16z Show on Spotify

Listen to the a16z Show on Apple Podcasts

Follow our host: https://twitter.com/eriktorenberg

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Discover more and add this to your library.

What does Rich Sutton’s "Bitter Lesson" reveal about the decisions Tesla is making in its pursuit of autonomy?

For more insights, check out Erin’s articles linked below.

Resources:

Find Anj on Twitter: https://x.com/anjneymidha

Find Erin on Twitter: https://x.com/espricewright

Read Erin’s article ‘A Software-Driven Autonomy Stack Is Taking Shape’: https://a16z.com/a-software-driven-autonomy-stack-is-taking-shape/

AI for the Physical World: https://a16z.com/ai-for-the-physical-world/

Stay Updated:

Let us know what you think: https://ratethispodcast.com/a16z

Find a16z on Twitter: https://twitter.com/a16z

Find a16z on LinkedIn: https://www.linkedin.com/company/a16z

Subscribe on your favorite podcast app: https://a16z.simplecast.com/

Follow our host: https://twitter.com/stephsmithio

Stay Updated:

Find a16z on YouTube: YouTube

Find a16z on X

Find a16z on LinkedIn

Listen to the a16z Show on Spotify

Listen to the a16z Show on Apple Podcasts

Follow our host: https://twitter.com/eriktorenberg

Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Key Takeaways

1
Elon Musk unveiled ambitious plans for Tesla's CyberCab, RoboVan, and a humanoid robot, showcasing their commitment to robotics and autonomy.
2
Rich Sutton's 'Bitter Lesson' suggests that general methods and abundant data will outperform specialized techniques in the development of AI.
3
Humanoid robots, while not economically pivotal in the short term, play an essential role in shaping public perception and excitement around robotics.
4
Integration between hardware and software in robotics presents significant challenges, yet also opens up opportunities across various under-invested industries.
5
Access to diverse and high-quality data is critical for building effective robotics systems, with various strategies being explored for data collection.
6
The successful integration of hardware and software cycles will determine the future of robotics, enabling more adaptive solutions in the industry.

Sponsors, Brands & Products

a16z

The content here is for informational purposes only and is not directed at any investors or potential investors in any a16z fund.

Tesla's Wii Robot Event: A Glimpse into the Future

Elon Musk and the Tesla team recently held their Wii Robot event, unveiling plans for innovative products like the unsupervised CyberCab, the RoboVan, and the Optimus humanoid robot. While critics argue the presentation lacked concrete details, supporters see it as an encouraging sign of Tesla's commitment to long-term goals in AI and autonomy. The discussion reflects changing perceptions around robotics and the increasing intersection of software and hardware, highlighting the importance of general-purpose approaches over specialized methods.

The Bitter Lesson and Autonomy Decisions

The podcast explores Rich Sutton's concept of the 'Bitter Lesson', which suggests that leveraging computational power and data is more effective than trying to engineer specific algorithms for AI tasks. The conversation emphasizes how Tesla's strategic focus on a deep learning pipeline aligns with this philosophy, prioritizing the accumulation of data and general algorithms to address the complex problem of self-driving vehicles. This approach could set the stage for significant advancements in autonomy over the coming years.

Human Connection through Robotics: Potential and Perception

An interest in humanoid robots, like Tesla's Optimus, is raised, particularly due to their ability to foster an emotional connection with humans. Despite skepticism about the necessity of humanoids in economic contexts, their representation aligns with the intriguing vision Musk presents, reinforcing the aspirational aspects of robotics. The conversation discusses the significance of human-like design in garnering public interest and acceptance, even if humanoids may not directly drive economic growth in the immediate future.

Challenges and Opportunities in Hardware-Software Integration

The intersection of hardware and software in robotics presents unique challenges, including skills shortages and the complexities of deploying solutions in real-world environments. The dialogue acknowledges a lack of standardized tools and resources for developers, emphasizing the need for integration across disciplines. As industries begin to explore the potential of software-driven hardware, opportunities arise in areas such as energy, manufacturing, and defense, which traditionally under-invested in technology.

The Data Dilemma in Robotics

Data is identified as the most crucial component for developing effective autonomous systems. The podcast discusses various methods for data collection, including simulation and crowdsourcing, as well as companies like Tesla that benefit from extensive data capture through real-world deployments. This highlights the necessity for diverse data sources to enhance model training, especially for companies that lack Tesla's integrated environments.

Decoupling Hardware and Software Cycles

The podcast highlights the distinction between software-focused and hardware-focused companies, emphasizing that successful ventures often commoditize hardware while maintaining innovation in software. Establishing a relationship between both sectors that enables quicker cycles and greater flexibility is a pressing challenge. The discussion concludes with optimism around the ongoing advancements in data-driven autonomy and the potential for transformative solutions across various sectors.

matches

Data is absolutely eating the world. What is good enough? We used to have the Turing test, which obviously were blown past now. His short hand for it was like the AWS AI.

He's got this idea of this distributed swarm of unutilized inference computers. Whether that's an oil rig, whether that's a mine, whether that's a battlefield, there's so many different use cases for a lot of this underlying technology that are really starting to see the light of day. It's basically an if not a win. In inevitability.

Early this month, Elon Musk and the team at Tesla held their Wii Robot event, where they unveiled their plans for the unsupervised, full-self-driving, CyberCab and RoboVan. Plus Optimus, their answer to consumer-grade humanoid robots, and also what Musk himself predicted would be, quote, the biggest product ever of any kind. Now, of course, none of these products are on the market yet, but several demos were on show at the event. Naturally, the response was mixed.

Sporters said we got a glimpse of the future, while critics said the details were missing. But in today's episode, we're not here to debate that. What we do want to talk about is what this indicates about the intersection of where hardware and software meet. So what does Rich Sutton's 2019 blog post, The Bitter Lesson, tell us about the decisions that Tesla is making in autonomy?

And how realistic is the quoted $30,000 price range? Also, there are the different layers of the autonomy stack, and where do we get the data to power it? And what does any of this look like when you exit the consumer sphere? We cover all this and more with A16C partners, Anjane Mita, and Aaron Price Wright.

Anjane previously founded Ubiquity 6, a pioneering computer vision and multiplayer technology company that sat right at this intersection of hardware and software, and was eventually acquired by Discord. Aaron, on the other hand, invests on our American Dignism team, with a focus on AI for the physical world. And if you'd like to dig even deeper here, Aaron has penned several articles on the topic that we've linked in our show notes. All right, let's get to it.

As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z.com slash Disclosures. So last week Tesla had their Wii robot event and Musk announced the CyberCub, the RoboVan, or as he liked to call it, the RoboVan, and the Optimus.

You guys are so immersed in this hardware software world. I'd love to just get your initial reaction. From my perspective, it wasn't that there was anything in particular that was super surprising, but what was exciting as just sort of a culmination of one thing that Elon Musk does really well and Tesla has done really well, which is continue to pour love and energy and money and time into a dream and a vision that's been going on for a really long time. like well passed when most financial investors and most people kind of lost the luster of self-driving cars after their initial craze in the mid to late 2010s and they've just continued to plot along and to continue to make developments and now we're finally seeing like this glimpse of the future for the first time in a really long time.

I think that's right. I think it was very impressive but unsurprising. Yeah. So I think the two schools of thought when people watched the event was Because one was absolutely this whole, oh my God, this is such vaporware.

He shared literally nothing on engineering details. What the hell? Come on, give us the meat on timelines and dates and prices. And then the opposing view was like, holy shit, they're still going.

They haven't given up on any of this autonomy stuff that he's been talking about for years. And I'm absolutely more empathetic towards the view that it was a lean towards the latter, which is that it saw it as an homage to the bitter lesson. It's sort of amazing blog post that I'm going to do a terrible job of summarizing by this great computer scientist Rick Sutton, which basically says that over the last 70 years or so of computer science history, what we've learned is that general-purpose methods basically beat out any specific methods in artificial intelligence in particular. Basically the idea that if you're working on solving a task that requires intelligence, you're usually better off leveraging Moore's law and more compute and more data than trying to hand engineer a technique or set of algorithms to solve a particular task.

And broadly speaking, that's been the big grand debate in self-driving and autonomy, I would say, for the last two decades, right? Yes, it's the sort of general purpose, bitter lesson school versus the, let's model self-driving as a specific task. As a set of discrete decision making algorithms. Right.

Unconnected to each other, right? A system to solve, let's say, edge detection around stop signs, right? Well, self-driving is a really hard problem. And you could totally say, well, there's so many edge cases in the world that we should map out each of those edge cases.

and I think it was an homage to the bitter lesson. So that's what I was most excited about is he did share actually details that their pipeline is basically an end to end deep learning approach. Which is incredible and probably true only for the last, my guess is 18 to 24 months. Right, yeah.

And I mean, in the bitter lesson, he also talks about the fact that it's really appealing to do the opposite because in the short term, you will get the benefit, but the broader deep learning approach ends up winning out in the long term and a lot of people talk about Musk, Musk says about himself that the timeline sometimes are off, but he's basically banking on that premise in the long term. It's basically an if not a win. In inevitability. Yes.

And I think the event was the first time that it really did feel in an emotional sense for the average American consumer. I'm not talking about the super duper tech literate people who wanted the details of the underlying models and their weights, but like for the average American consumer, the first time that this version of the feature felt like an inevitability. And before we get into maybe the specifics around where else hardware and software are intersecting, I'd love to just talk about that, that average person who's watching because you guys are meeting with companies and investors and this has been going on for quite some time. So I'm just curious if maybe you noticed anything under the hood or maybe the meta in that announcement or event, that maybe the average person watching is, you know, what are they seeing?

They're saying things like, oh, maybe it was human controlled and not like fully AI and device or other people are commenting on the fact that these humanoids are shaped like humans. Like, why do we need that? On the topic of humanoids, I think humanoids are a great choice of embodiment for a robot to really emotionally connect to and speak to a human being watching because I can relate to a human form factor. obviously we found out that it was teleoperated, which is, in my opinion, still doesn't take away from how cool and amazing it was.

The human form factor is a way to connect what is happening with robotics to a regular person who was like, okay, yes, I like see myself in that. This looks like Star Wars or some other sci-fi movie. In reality, maybe this is a controversial opinion. I don't see the vast majority of economic impact over the next decade from robotics coming from the humanoid form factor, but that doesn't take away from the power of the symbol of having a humanoid make a drink at this event because it just like connects back to the sort of science fiction promise of our childhoods getting sort of finally delivered.

The opening sequence, he started with like a sci-fi, I think it was a blade runner visual, and he was like, we all love sci-fi, and I want to be wearing that jacket that he's wearing in the picture, but we don't want any of the other dystopian stuff. And so that definitely stuck out to me is that he did not start the way he usually does. It's often a technical first sort of story, but he started with, here's a vision for where I think the world should go. So it was much more Disney-esque than that, and it was quite poetic.

I think we literally did it on the Warner Brothers studio lot. And so they like recreated a bunch of cities, and I think they had on-site at the event, the Robo Vans, taking people around from these simulated cities. There was a sort of theatricality to it all that stuck out to me, which I thought was quite different and I thought it was refreshing because the core problem with this branch of AI which is largely deep learning based and bitter lesson based is that it's an empirical field unlike call it Moore's Law which was predictive where you basically know if you double the number of transistors you get this much more performance on the chip and it's just about pure execution AI is much more empirical you don't really know when the model is going to get done training and when it does get training whether it will converge or not or even what is converging like what is good enough. We used to have the turning test, which obviously we're blown past now.

It's a feeling more than it is a set of discrete metrics that you can really point to. Right. So it made a lot of sense to me that he's trying to decouple this idea of progress from a specific timeline because I just think we're setting ourselves up for every time. You ask a deep learning researcher.

So when's that GPT-5 model going to show up? It's like the most frustrating question ever, right? Because they don't know. We don't know.

And frankly, Usually sometimes they show up earlier than schedule and show up later, and by the way, you can look at the stock market's reaction to prime example of how people have been so conditioned by, I would say, the Steve Jobsian, Apple, like, Kaden's year-on-year of, like, here's your new iPhone. It's just incremental, but predictable, I think forecasting that the tech industry keeps trying to reward. And I think what he's doing is pretty refreshing, which is saying, look, here's a vision for where we want to go, but it's decoupled. The second thing on the humanoid piece that I was quite impressed by is actually the quality of the teleoperation.

So everybody's talking about how, oh, this is fake, this is all smoke and mirrors, it's just people. Hell operation is real. I'm just going to say. Why is no one talking about that?

Have you ever tried? I mean, I've tried. It's so hard. I like, we were at a company two weeks ago and they've got these teleoper robots and the founder was demoing a mechanical arm that he was teleoperating with a gamepad and he was folding clothes with it and I was like, oh, that looks simple, he's like, here, try it.

It was one of the hardest manipulation things I've ever tried. And by the way, we tried that with VR headset, with six-dolph motion controllers, almost harder to do, teleoperating something especially over the internet in a smooth fashion with precision is incredibly hard. And I don't think people appreciate the degree to which they've really solved that pipeline. Yeah, I was actually really impressed by that.

And I think that there's huge opportunity for teleop in sort of production applications that will have like massive economic benefit. Even before we have true robots running around managing themselves. Because if you think about there's all these really hard and really dangerous or hard to get to jobs or there's labor, differentials where it's a lot harder to hire people to do certain things in certain locations. And if we can imagine a future where the teleop that we saw last week at the event is something that's like widely available, that's incredible.

Imagine not having to go and service a power line, but you can actually teleop a robot to do that for you, but still have the level of sort of human training and precision needed to make a really detailed and specific evaluation. The promise of that is really cool even before we get to robots. So that was really exciting. Yeah, it's like a stop along this journey.

And so if we talk about that journey, the arc of hardware and software coming together and maybe a different way than we've seen in the past, just as an example. So Mark famously said software is eating the world that was in 2011, we're in 2024. And it does feel like the last decade has been a lot of traditional software, not so much integrating with the physical world around us. And so where would you place us in that trajectory?

Because we're seeing it with autonomous vehicles, but I get the sense that's not the only place where this is happening. Yeah, this is where I spend 95 percent of my time and all of these industries that are just starting to see the glimmers of what autonomy and sort of software-driven hardware can bring. What's really interesting is just actually a dearth of skills of people who know how to deal with hardware and software together. You have a lot of people that went and got computer science degrees over the last decade and relatively speaking a lot fewer that went and got electrical engineering or mechanical engineering degrees and we're starting to see the rise of oh shoot we actually need people who understand not just maybe how the software works in the cloud with Wi-Fi where you have unlimited access to compute and you can and retry things as many times as you want, and you can ship code releases all day every day, but you actually have kind of a hardware deployment, where you have limited compute in an environment where you maybe can't rely on Wi-Fi all the time, where you have to tie your software timelines to your hardware production timelines.

Like these are a really difficult set of challenges to solve, and right now, like there just isn't a lot of standardized tooling for developers and how to do that. So it's interesting. we're starting to see portfolio companies of ours across really different industries that are trying to use autonomy, whether it's oil and gas or water treatment or HVAC or defense. They're like sharing random libraries that they wrote to connect to like particular sensor types because there's not this like rich ecosystem of tooling that exists for the software world.

So we're really excited about what we're starting to see emerge in the space. Even Elon said when he's talking about these two different products that he's unveiling, right, Optimus, and then you have the RoboVans or Cybercabs. And those seem like two completely different things, but he even said in the announcement, he said, everything we've developed for our cars, the batteries, power electronics, advanced motors, gearboxes, AI inference computer, it all applies to both. Right, so you're seeing this over there exciting.

When I was watching it, I was just nerding out because my last company was a computer vision, 3D mapping and localization company. So I unfortunately spent too much of my life calibrating LiDAR sensors to our computer vision sensors, So because our whole thesis when I started back in 2017 was that you could do really precise positioning just off of computer vision and that you didn't need fancy hardware like light hours or depth sensors. And to be honest, not a lot of people thought that we could pull it off. And frankly, I think there were moments when I doubted that too.

And so it was just really fantastic to see that his bet and the company's bet on computer visions and a bunch of these sensor fusion techniques that would not need specialized hardware would ultimately be able to solve a lot of the hard navigation problems which basically means that the way they're solving precision is instead of throwing more sensors on the car is to basically throw more data at the problem. And so in that sense data is absolutely eating the world. And you asked where on the trajectory are we of operating the world? And I think we're definitely on an exponential that has felt like a series of stacked sigmoids.

Often it feels like you're on a plateau. But a series of plateaus totally make up an exponential if you zoom out enough. And earlier in the conversation we talked about the better lesson, a number of other teams in the autonomy space decided to tackle it as a hardware problem, not a software problem, right? Where they said, well, more LiDAR, more expensive LiDAR, more GPUs, more sensors, right?

And Elon's like, you know, actually, I want cheap cars that just have computer vision sensors. And what I'm going to do is use a bunch of the custom really expensive sensors that many other companies put on the car, which is at inference time, and he's just going to use them at train time. So Tesla does have a bunch of like really custom hardware that's not scalable that drives around the world in their parking lots and simulation environments and so on. And then they distill the models they train on that custom hardware to a test-time package.

And then they send that test-time package to their retail cars, which just have computer vision sensors. And the reality is that's a raw arbitrage, right, between sensor stack. And it allows the hardware out in the world to be super cheap. The result there is software is eating the sensor stack out in the world that makes the cost of these cars so much cheaper that you can have a $30,000 fully autonomous car versus $100 plus $1000 of cars that are fully loaded with these lidar sensors and so on.

But I think in order to have the intuition that you can even do that, you really actually have to understand hardware. If you just understand software and hardware is like a sort of a scary monster that lives over here and maybe you have a special hardware team that does it, it's going to be hard for you to have the confidence to say, no, we can do it this way. I think you're totally, which is that the superpower that Tesla has is his ability to go full stack, right? Because a lot of other industries often segment out software versus hardware like you're saying and that means that people working on algorithms and the autonomy part just treat hardware as like an abstraction, right?

You throw over a spec, it's an API, it's an interface that I program against and I have no idea what's going on. You don't have to worry about the details. You don't have to worry about it. Which part of it is super powerful.

It's unlocked this whole general purpose wave of models like chat GPT and so on, right? Because it allows people who specialize in software to not have to think about the hardware. It's also what's driven sort of the software renaissance of the last 15 years. Absolutely, decoupling, right?

Composition and abstraction is sort of the fundamental bases of the entire computing revolution. But I think when you're like him and you're trying to bring a new device to market, kind of like what Jobs did with the iPhone down by going full stack, you end up unlocking massive efficiencies of cost. And I think that's what this event may have been lost in the sort of theatricality of it all, is the fact that he's able to deliver an autonomous device to retail consumers at a cost profile through vertical integration that would just not be possible if it was just a software team buying hardware from somebody else and building on top. Can we talk about those economics, by the way, just attacking that head on?

and both Optimus and CyberCab were quoted as being under the 30K range. Is that really realistic? And then tied into what you were saying, we see other autonomous vehicles which are betting more on the LiDAR and the sensors, which also have come down in price pretty substantially. My guess is Elon is backing into the cost based on what people are willing to pay.

And he will do whatever it takes to get those costs to line up. I mean, it's the same thing he did with SpaceX. He will operate within whatever cost constraints he needs to operate within even if the rest of the market or the research community is telling him it's not possible. Well, obviously like a 30K humanoid robot is way less than what most production industrial robotic arms cost today, which I think are more in the 100K range for the ones that are used in like the high end factory.

So if you can get it down to 30K, that's really exciting. I also don't necessarily think you need even a 30K humanoid robot to accomplish a wide swath of the automation tasks that would pretty radically transform the way our economy functions today. Yeah, I think Aaron's writing that there's probably a top-down director, which is do whatever it takes to get into the cost footprint This car has to cost 30k Right, but I think if you do a bottoms-up analysis I don't think you end up too far because actually if you just break down the kind of bomb on a Tesla Model 3 You're not dramatically far off from the sensor stack you need to get to a 30,000 dollar car, right? This is the beauty of solving your hardware problems with software is you don't need a $4,000 scanning lighter on the car.

So I think on the cyber cab, I feel much more confident that the cost footprint is gonna fall in that range because frankly, we kind of have at least an ancestor on the streets, right? The thing that gets prices up is custom sensors because it's really expensive to build custom sensors in short production runs. And so you either have scale of manufacturing like an Apple and you make a new CMOS sensor or a new Face ID sensor and you get cost economies of scale because you're shitting more like 30 million devices at your first run. Or you just lean on commodity sensors from the past era and if you tackle most of your problems in software, which is what he's doing.

And to that point, when he's betting on software, another interesting thing that he announced was really over-specking these cars to almost change the economics potentially, based on the fact that those cars could be used for distributed computing. To your point, Ange, if you put a bunch of really expensive sensors on the car, you can't really distribute the load of that in any other way than driving the car, right. But if you actually have this computing layer that's, again, in his case, he's saying he's planning to overspec, that actually can fundamentally change like what this asset is. And you kind of saw the same thing even with Teslas today where he's talking about this distributed grid, right?

All of a sudden these large batteries are being used not just for the individual asset. So do you have any thoughts on that idea or if we've seen that elsewhere? He was a bit skimpy on details in that. But I think He did say that the AI5 chip is over-spect.

It's probably going to be four to five times more powerful than the HW4, which is their current chip. It's going to draw four times more power, which probably puts it in that like 800 watts of storage, which for context, you know, average hair dryer is at about 1,800 watts. I mean, it's hard to run power on the edge. But I think what he said was something to the effect of like, your car's not working 24 hours a day.

So if you're driving, call it eight hours a day in LA traffic, God bless whoever's having to do that. For real. Hopefully they're using self-driving. One would hope.

Actually, if you open that, his bitch would want a story about driving to El Segundo and he's saying you can fall asleep and wake up on the other side. But I think the T-shirt size he gave was about a hundred gigawatts of unallocated inference compute just sitting out there in the wild. And I think his shorthand for it was like the AWS of AI, right? He has got this idea of this distributed swarm of unutilized inference computers.

And it's a very sexy vision. I really want to believe in it. Ground us on. Is this realistic?

Well, I know, I think it's realistic for workloads that we don't know yet in the following sense, right? That the magic of AWS is that it's centralized and it abstracts away a lot of the complexity of hardware footprints for developers. And by centralizing almost all their data centers and single locations with very cheap power and the electricity and Cooling these clouds are able to pass on very cheap inference costs to the developer now What he's got to figure out is how do you compensate for that in a decentralized fashion? And I think we have kind of prototypes of this today like they're these vast decentralized clouds I think one is literally called vast of people's unallocated gaming rigs people have millions of in video 4090 gaming cards sitting on their desks that aren't used and And historically, those have not yet turned into great businesses or high-utilized networks because developers are super sensitive to do things cost and reliability.

And by centralizing things, AWS is able to ensure very high uptime and reliability, whereas somebody's GPU sitting on their... Maybe available, maybe they're driving to El second row. And there are just certain things, especially with AI models that are hard to do on highly distributed compute where you actually need good interconnect and you need things to be reasonably close to each other. Maybe in his vision, there's a world where you have optimist robots in every home.

And somehow your home optimist robot can take advantage of additional compute or additional inference with your Tesla car that's like sitting outside in your driveway. Who knows? Right. Okay.

Well, this event clearly was focused on different models that are consumer facing. So again, cybercab that's for someone using an autonomous vehicle. Optimus is a humanite robot probably in your home. But Aaron, you've actually been looking at the hardware software intersection in a bunch of other spaces, right?

And as you alluded to earlier, maybe different applications with better economics at least today. I think long-term there's no market bigger than the consumer market. So everyone having a robot in their home in a Tesla car in their driveway that's also a Robotaxi has huge economic value, but that's also a really long term vision. And there's just so much happening in autonomy that's taking advantage of the momentum and the developments that companies like Tesla have put forward into the world over the last decade that actually have the potential to have meaningful impact on our economy in the short term.

I think the biggest broad categories for me are largely the sort of dirty and unsexy industries that have very high cost of human labor often because of safety or location, access, whether that's when oil rig out in the middle of Oklahoma somewhere that's three hours drive from the New York City, whether that's a mine somewhere in rural Wyoming, that freezes over for six months out of the year so humans can't live there and mine, whether that's a battlefield where you're starting to see autonomous vehicles go out and clear bombs and mines from battlefields to protect human life. There's so many different use cases for a lot of this underlying technology that are really starting to see the light of day, so very excited about that. And as we think about that opportunity, you've also talked about this software-driven autonomy stack. So as you think about the stack, what are the layers, can you just break that down?

Yeah, sure. So, across whether it's a self-driving car or sort of an autonomous control system, we're seeing the stack break down into pretty similar categories. So first is perception you have to see the world around you know what's going on be able to see if there's a trash can be able to understand if there's a horizon if you're a boat. The second is something on knows really well, which is location and mapping.

So okay, what do I see? How do I find out where I am within that world based on what I can see and what other sensors I can detect, whether it's GPS, which often isn't available in battlefields or in warehouses, etc. the third is planning and coordination. So that's, okay, how do I take a large task and turn it into a series of smaller tasks?

So what is more of an instant reaction? I don't have to really think about how to take a drink of water, but I might have to think about how to make a glass of lemonade from scratch. So how do I think about compute across those different types of regimes when something as more of an instinct, first is when something has to be sort of taken down and processed into discrete operations. And then the last one is control.

So that's like, how does my brain talk to my hand? Like, how do I know what are the nerve endings doing in order to pick up this water bottle and take a drink out of it? And that's a really interesting kind of field that's existed for decades and decades, but for the first time probably since the 70s, we're starting to see really interesting stuff happen in the space of controls around autonomy and robotics. And I would say like all of these are pre-existing areas.

None of this is wildly new, but I think in the last two years, especially with everything that's happening with deep learning, video language models, broadly speaking, AI has pushed every single one of these kind of to its limit and to a new state of the art. And there just aren't tools that exist to tie all that together. So every single robotics company, every single autonomous vehicle company is basically like rebuilding this entire stack from scratch, which we see as investors as a really interesting opportunity as the ecosystem evolves. And as you think about that ecosystem, people kind of say that as soon as you touch hardware, you're literally working on hard mode compared to just a software-based business.

So what are the unique challenges, even with maybe that AI wave today that's pushing things ahead? How would you break down what becomes so much harder? I think I'm touched on this a little bit before, but the more you can commoditize the hardware stack, the better. So, the most successful hardware companies are the ones that aren't necessarily inventing a brand new sensor, but are just taking stuff off the shelf and putting it together.

But still, tying everything together is really hard. When you think about releasing a phone, for example, Apple has a pretty fast shipping cadence, and they're still releasing a new phone only every once a year. So you have to essentially tie a lot of your software timelines to hardware timelines in a way that doesn't exist when you can just sort of ship whenever you want into cloud. If you need a new sensor type or you need a different kind of compute construct or you need something fundamentally different in the hardware, you're bound by those timelines, you're bound by your manufacturers availability, you're bound by how long it takes to quality engineer and test a product, you're bound by supply chains, you're bound by figuring out how these things have to integrate together.

So the cycles are often just quite a lot slower. And then the other thing is when you're interacting with the physical world, you get into use cases that touch safety in a really different way than we think about with pure software alone. And so you have to design things for a level of like hardiness and reliability that you don't always have to think about with software by itself. If your chat GPT is a little slow, it's fine, you can just try again.

But if you have an autonomous vehicle that's like driving a tank on a battlefield autonomously and something doesn't work, like you're kind of screwed. So you have to have a much higher level of rigor and testing and safety built into your products, which slows down the software cycles. The holy grail is sort of general purpose indulgence for robotics, which we we still don't have when you train a general model. You basically get the ability to build hardware systems that don't have to be particularly customized.

And that reduces hardware iteration cycles dramatically. Because you can basically say, roughly speaking, these are the four or five commodity sensors you need, the smarter your brain, so to speak, the less specialized your appendages have to be. And I think what a number of really talented teams are trying to solve today is, can you get models to generalize across embodiments? right?

Can you train a model that can work seamlessly on a humanoid form factor or a mechanical arm, a quadruped, whatever it might be? And I'm quite bullish that it will happen. I think the primary challenge there that teams are struggling with today is the lack of really high quality data. Right?

The big unknown is just how much data, both in quantity and quality, do really need to get models to be able to reason about the physical world spatially in a way that abstracts across any hardware. I'm completely convinced that once we unlock that, the applications are absolutely enormous, right? Because it frees up hardware teams, like Aaron was saying, from having to couple their software cycles from hardware cycles. It decouples those two things.

And I think that's the holy grail. I think what Tesla, the victory of the autonomy team over there is having realized eight years ago, the efficacy of what we call early fusion foundation models, right? Which is the idea that you take a bunch of sensors in at training time and different inputs of vision, depth, you take in video, audio, you take a bunch of different six-dough sensors and you tokenize all of those and you fuse them all at the point of training and you build an internal representation of the world for that model. In contrast, the LLM world does what's called late fusion.

You often start with a language model, let's train just on language data and then you duct tape on these other modalities, right, like image and video and so on. And I think the world has now started to realize that early fusion is the way forward, but of course they have an eight-year head start. And so I get really excited when I see teams either tackling the data sort of challenge for general spatial reasoning or teams that are taking these early fusion foundation model approaches to reasoning that then allow the most talented hardware teams to focus really on what they know best. Where are these companies getting training data?

Because you mentioned Tesla for example, yes we've had cars on the road, tons of them with these cameras and sensors. I still think that one of the smartest things Elon did was turn on full self-driving for everybody for like a month-long trial period last summer. I have a Tesla and I turned it on for my free months and it was like a life-changing experience using and I obviously couldn't get rid of it and so now not only do I now pay for full self-driving but I also I'm giving him all my data. So to me that's really clever and so I'm curious if you talk about some these other applications, do they have the number of devices, or in this case, cars for Tesla capturing this data?

Or how else are we going to get this spatial data? This is the big race in robotics right now. I think there are several different approaches. Some people are trying to use video data for training.

Some people are investing a lot in simulation and creating digital 3D world. And then there's a mad rush for every kind of generated data that you could possibly have. So whether that's robotic teleoperated data, whether that's robotic arms and offices, most of these robotics companies have pretty big outposts where they're collecting data internally. They're giving humanoids to their friends to have in their homes.

It's a yes and scenario right now where everyone is just trying to get their hands on data literally however they can. I think it's the Wild West. But if you're Tesla, then the secret weapon you have is you've got your own factories, right? So the Optimus team has a bunch of humanoid's walking around the factories, constantly learning about the factory environment.

And that gives them this incredible self-fulfilling, sort of compounding loop. And then, of course, he's got the testically like Aaron was seeing earlier with FSD. I'm proud to have been a month one subscriber for it. And I'm happy that I'm contributing to that training cycle, because it makes my model like smarter next time around.

So the challenge then is for companies that don't have their own full stack, sort of fully integrated environment, where they don't have deployments out in the field and Darren's point, you can either take the simulation route for that and say we're going to create these sort of synthetic pipelines. Or we're seeing this, yeah, huge build out of teleop fleets like with language models, you had people all around the world in countries showing up and labeling data. You have teleop fleets of people bilating mechanical arms hardware around the world. I think there's an interesting sort of third new category of efforts we're tracking, which is crowdsourced coalitions, right?

So an example of This is the DeepMind team put out this maybe a year and a half ago robotics data set called RTX where they partnered with a bunch of academic labs and said, Hey, you send us your data. We've got compute and researchers. We'll train the model on your data and then send it back to you. And what's happening is just different labs around the world who have different robots of different kinds, summer arms, summer quadruped summer bipeds.

And so instead of needing all of those to be centralized in one place. There's a decoupling happening where some people are saying, well, we'll specialize in providing the compute and the research talent and then you guys bring the data and then it's a gift to get model, right? Which we saw in some cases with the internet early on and videos an example of this where their research team isn't it's stacking a bunch of robots in house So they instead partnering with people like Farmer labs who have arms doing pipetting and wet lab experiments and saying you send us the data We've got a bunch of GPUs. We got some talented deep learning folks. We'll train the model send it back to you I think it's an interesting experiment, and there's reason to believe this sort of gift-to-get model might end up actually having the highest diversity of data.

But we're definitely in sort of full experimentation land right now. Yeah, and my guess is we'll need all of it. So it sounds like data is a big gap, and it sounds like some builders are working on that. But where would you guys like to see more builders focused in this hardware software arena?

Especially because I do think there are some consumer facing areas where people are drawn to, they see an event like this, and they're like, oh, I want to work on that. But yeah, I'm pretty excited about the long tail of really unsexy industries that have outsized impact on our GDP and are often really critical industries, where people haven't really been building for a while, things like energy, manufacturing, supply chain, defense. These industries that really carry the US economy, where we have under-invested from a technology perspective probably in the last several decades are poised to be pretty transformed by this sort of hardware software melding in autonomy. I'd love to see more people there.

I'm very excited for all the applications there and talked about, and I think to unlock those, we really need a way to solve this data bottleneck, right? So startups, builders who are figuring out really novel ways to collect that data and the world get it to researchers, make sense of it, curate it. I think that's sort of a fundamental limiter on progress across all of these industries. We just need to sort of 10x, the rate of experimentation in that space.

Alright, that is all for today. If you did make it this far, first of all, thank you. We put a lot of thought into each of these episodes whether it's guests, the calendar Tetris, the cycles with our amazing editor Tommy until the music is just right. So, if you like what we put together, consider dropping us a line at ratethispodcast.com slash a16c and let us know what your favorite episode is.

It'll make my day and I'm sure Tommy's too. We'll catch you on the flip side.","created_at":"2024-10-24T10:01:27.000000Z","updated_at":"2025-07-15T07:04:30.000000Z","requested_by":[2],"status":"completed","delay_time":null,"worker_id":null,"job_id":null,"execution_time":null,"runpod_job_id":null,"error_message":null,"completed_at":null,"word_level_timestamps":null,"claimed_at":null,"retry_count":0}

Ask questions about this episode