The journey to massive scale and ultra-resilience

Key takeaways

Open source has revolutionized how quickly and openly developers can adopt and innovate with new technologies.
There are significant technical and business challenges in building massive, resilient distributed databases, but technologies like consensus algorithms and cloud-native architectures are paving the way.
CockroachDB was inspired by Google's Spanner and aims to offer a resilient, open-source, cloud-native SQL database that can operate at web scale.
To compete with the giants of cloud services, companies need to focus on innovation, multi-cloud flexibility, and user-friendly, cost-effective services.
The future of data infrastructure involves global, low-latency systems that provide seamless experiences; developers should prepare for an environment where databases are serverless, geo-distributed, and accessible to all scales of users.

Summary

Open-Source Foundations and Early Career

Spencer Campbell discusses his early involvement in open source software, notably co-creating the GIMP in the 1990s, which grew beyond his and his co-founders' direct involvement. Moving from graphics to databases, he highlights the importance of open source as a catalyst for rapid, community-driven development, and how his career path led to a focus on building resilient, scalable databases. His time at Berkeley and early tech ventures shaped his understanding of software's potential and limitations.

The Evolution of Database Technology and Challenges

The conversation dives deep into the technical evolution of databases, emphasizing the challenges of sharding and scalability. Spencer explains the advancements made by Google with technologies like Bigtable and Spanner, which inspired CockroachDB. They discuss the necessity for databases to handle web-scale data, especially as IoT and API-driven applications grow. The importance of consensus algorithms like Raft and Paxos for strong consistency and resilience in distributed systems is also covered.

Birth and Development of CockroachDB**

Spencer shares how the idea for CockroachDB originated from a desire to create a resilient, open-source database inspired by Google's Spanner. The project was initially a passion and side project during his time at Square, motivated by the need for highly resilient and scalable databases. It was built with concepts like cooperative nodes and well-understood protocols, aiming for resilience akin to cockroaches' survival ability. The project gained traction, leading to venture funding and its ongoing development as a commercial product.

Business Strategy and Market Positioning**

The discussion covers Cockroach Labs' approach to competing with large cloud providers like AWS, Google Cloud, and Azure. Spencer emphasizes the importance of out-innovating, leveraging multi-cloud and hybrid deployments, and building a database that is fully cloud-agnostic, serverless, and affordable. The vision includes making databases infinitely scalable and accessible, reminiscent of email services like Gmail, with free tiers for developers to encourage adoption.

Future of Data Infrastructure and Developer Experience**

Looking ahead, Spencer highlights how next-generation applications will require ultra-low latency and global operations to deliver seamless user experiences. The integration of new infrastructure, 5G, and global data architectures will empower developers to build powerful, real-time, geo-replicated systems. Cockroach Labs aims to partner with cloud providers and other tech firms to develop these resilient, scalable stacks, making building global apps as straightforward as local ones.

Show notes

This week Adam talks with Spencer Kimball, CEO and Co-founder of Cockroach Labs — makers of CockroachDB an open source cloud-native distributed SQL database. Cockroach Labs recently raised $160 million dollars on a $2 billion dollar valuation. In this episode, Spencer shares his journey in open source, startups and entrepreneurship, and what they're doing to build CockroachCloud to meet the needs of applications that require massive scale and ultra-resilience.