Building CI for the age of AI Agents with Aayush Shah

Key takeaways

Blacksmith is focused on optimizing CI workflows to handle the increasing complexities and demands brought by AI coding agents.
The platform aims to improve visibility into test failures and streamline resource allocation through innovative infrastructure strategies.
Multi-tenancy plays a crucial role in Blacksmith's ability to manage workload spikes efficiently while easing resource burden.
A hands-on experience from previous companies informs Blacksmith's engineering strategies and product development.
Observability and effective resource management are vital for successful CI, especially as team sizes and outputs grow.

Summary

Introduction to Blacksmith and CI Challenges

In this episode, host Alex interviews Aayush Shah, co-founder of Blacksmith, a platform designed to optimize Continuous Integration (CI) for GitHub Actions. They discuss how AI coding agents have notably influenced CI workflows, increasing demand and complexity. Blacksmith aims to help teams manage their CI more efficiently by providing enhanced visibility into test failures and improving response times to workload spikes.

Blacksmith's Unique Approach to CI Infrastructure

Aayush delves into the innovative approaches Blacksmith has adopted to build a specialized CI compute fabric. By operating outside traditional hyperscalers, they've optimized costs related to resource management and significantly improved performance for CI tasks. This includes avoiding the pitfalls of EBS volume dependencies that do not suit temporary CI jobs.

Multi-Tenancy and Workflow Management

They explore the importance of multi-tenancy in CI workloads, especially how aggregating customers can smooth out the peak demands faced during high-usage periods. Aayush explains how Blacksmith strategically positions itself to handle bursts in workloads and manage resource allocation efficiently, thus maintaining a high-quality service across varying customer demands.

The Future of CI with AI Integration

The discussion highlights the evolving landscape of CI, particularly influenced by AI tools and coding agents, and how teams are adapting to incorporate these advancements into their workflows. Aayush shares insights on the growing need for observability in CI to ensure code integrity while managing increased CI spikes from concurrent AI-driven tasks.

Engineering and Team Structure at Blacksmith

Aayush also touches on the team dynamics at Blacksmith, with a split between engineering and sales offices in New York and San Francisco. They discuss their tech stack, including a significant use of Go, TypeScript, and PHP, noting how using Laravel for their control plane enhances productivity due to the pre-solved problems it presents.

Show notes

Today's episode is with Aayush Shah. Aayush is one of the co-founders of Blacksmith, which is a CI compute platform. Basically, Blacksmith will run your GitHub Actions jobs faster and with more visibility with the standard GitHub Actions CI runners.

The founding team has a fun background doing systems work at Cockroach and Faire, and they're taking on a big problem in running this massive CI fleet.

The explosion in AI agents has really changed the CI world. CI is more useful than ever, as you want to be sure the changes from your agents aren't breaking your existing functionality. At the same time, there's a huge increase in demand and spikiness of CI workloads as developers can fire off multiple agents to work in parallel, each needing to run the CI suite before merging. Aayush talked about how they're handling this load and facilitating visibility into test failures.

We also covered cloud economics. Aayush said the traditional cloud-based storage options don't work for them -- EBS and locally attached SSDs are too expensive for their workloads where they don't need the standard durability guarantees. He walks us through building their own fleet outside the hyperscalers and the plans going forward, along with some of the economics of multi-tenancy that Blacksmith has previously written about.