How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal

Ankur Goyal discusses how AI agents are revolutionizing traditional engineering tasks, especially in areas like database optimization and infrastructural experimentation. Using Codex, he automates long-term benchmarking experiments to evaluate different indexes, storage formats, and engines, significantly accelerating the process of technical innovation. He emphasizes that with AI, there's no longer a justified excuse to skip rigorous benchmarking since agents tirelessly perform these tasks, leading to higher quality outcomes and faster iteration cycles.

The conversation tackles the misconception held by some engineers that AI cannot handle complex, technical problems. Ankur counters this by sharing how AI, through continuous, rigorous testing and benchmarking, outperforms manual efforts on tedious, high-stakes tasks. Human attention spans decay over long, monotonous processes, whereas AI can sustain high-quality analysis consistently. Practical application demonstrates that AI improves both speed and quality, enabling teams to tackle much larger and more intricate problems than before.

Evals are presented as a modern, scalable alternative to traditional PRDs, focusing on defining success criteria in a quantifiable way. Ankur illustrates their use by building scoring functions for assessing AI responses to FAQs, and demonstrates how to run these evals live within a safe environment to iteratively improve model outputs. The key lesson is that evals shift the focus from manual, anecdotal quality checks to systematic, data-driven evaluation, scaling quality assurance beyond what humans can feasibly manage.

Ankur shares his approach to managing multiple foreground agents running in parallel across cloud environments, optimizing their configuration with cloud tools like EC2 and S3, and using tools like tmux for session management. He highlights the importance of structured workflows, investing in continuous integration, and dedicating focused periods for coding. He advocates chunking work into manageable pieces and leveraging AI to handle routine or complex long-tail tasks, thereby increasing overall velocity and reducing cognitive fatigue.

When AI agents fail or produce subpar outputs, Ankur emphasizes the importance of iterative correction—shutting down the session, revisiting and refining prompts, and rerunning evals. He advocates for building a feedback loop that captures failure modes and uses them to improve prompting strategies continuously. Ankur also shares insights from experience, including the benefit of writing eval scripts by hand when automated scripts get stuck or confused, to better understand and troubleshoot model behavior.

The episode concludes with practical advice for engineering teams on integrating AI effectively: prioritize building pipelines for continuous evaluation and data enrichment, focus on improving CI systems to accelerate velocity, and adopt an experimental mindset. Ankur stresses that investing in automation, rigorous testing, and structured workflows will be essential for teams to unlock the full potential of AI-assisted engineering and keep pace in a rapidly evolving landscape.