Open Source Self-Improving AI with Vignesh Baskaran

Key takeaways

Self-improving AI agents are achievable through a combination of touching both model weights and system harnesses, enabling better performance without human intervention.
Hexo Labs’ meta-agent Sia automates the evaluation and improvement of task-specific agents, utilizing proxy evals and scientific experimentation to validate improvements.
Formal, mathematical proof of self-improvement remains a theoretical goal; practical, empirical validation through rigorous testing and proxy evals is currently the most effective approach for commercial applications.
AI model deployment is moving toward a diversified ecosystem of multiple models and fine-tuned weights from various providers, balancing performance, cost, and latency based on specific use cases.
Automating routine validation, trace analysis, and experiment management is crucial for scaling self-improving agent development, saving time, reducing errors, and ensuring quality.

Summary

Introduction and Background of Vignesh Baskaran

Vignesh Baskaran shares his extensive experience in neural networks and AI, dating back to 2012. His journey from training neural networks, working as a data scientist, ML engineer, to becoming a CTO at Hexo Labs is driven by the fundamental goal of making systems behave as intended. His most notable project was building a litigation search engine that even Google adopted. Currently, his focus is on developing autonomous agents that can rewrite and retrain other agents without human intervention, emphasizing the importance of touching both the model weights and the system harness for optimization.

The Work of Hexo Labs and Self-Improving Agents

Hexo Labs is pioneering research in autonomous agents that can self-improve. Central to this effort is Sia, a meta-agent capable of evaluating and improving task-specific agents through automated testing, environment interactions, and proxy evaluations. The discussion highlights how these agents can optimize both their underlying model weights and their harness or tools, leading to better performance. Hexo’s methodology involves rigorous scientific experimentation and validation, even when ground truth data are scarce, using proxy evals and innovative techniques like the Proxy Evolution Agent. This approach draws inspiration from DeepMind’s Alpha series, emphasizing iterative, scientifically driven improvements.

Challenges and Techniques in Self-Improvement and Validation

Vignesh discusses the complexity of proving an agent's self-improvement either theoretically or empirically. Formal verification is too strict for practical, commercial AI applications; instead, empirical validation through rigorous testing and proxy evals is currently the pragmatic approach. He emphasizes the importance of scientific rigor, proper validation, and avoiding common pitfalls like overfitting or VAE evals, which can misrepresent agent quality. An example provided is a case where deep trace analysis uncovered a silently corrupted eval file—highlighting the importance of thorough validation and the power of automation tools in managing complex trace data.

Model Fine-Tuning, Multi-Model Strategies, and Performance Trade-offs

The conversation explores the future of AI model deployment, emphasizing the diversification of models from multiple providers, including open-source alternatives. Vignesh explains how Hexo’s approach enables simultaneous training and fine-tuning of multiple models and weights for different tasks, facilitating a multi-model ecosystem. The discussion covers how different use cases—ranging from high-performance, low-latency applications to cost-sensitive, routine tasks—dictate different optimization strategies. Hexo’s platform allows for scientific hyperparameter optimization, version control, and experimentation, helping enterprises balance performance, cost, and latency effectively.

Automating Routine Tasks and Future Directions

Vignesh emphasizes that most users and organizations seek automation of routine, mundane tasks to save time and improve reliability, rather than groundbreaking new algorithms. Sia aims to automate hyperparameter searches, validation processes, and trace analysis, empowering organizations to manage large-scale agent operations efficiently. Future improvements include dynamic knowledge base management, self-generating evaluation sets based on trace analysis, and enhanced support for proprietary data. The conversation concludes with a strong push toward open source tools, inviting listeners to explore Hexo’s GitHub repositories and website for more information.

Show notes

Today we are talking with Vignesh Baskaran, the CTO and co-founder of Hexo Labs, about teaching AI agents to improve themselves. Vignesh has been training neural networks since 2012, back when he was still called a data scientist.

Then he became an ML engineer and now an AI engineer, though he says the underlying work has never really changed. It's to figure out how to make a system behave the way you intend it to. He built the litigation search engine that Google itself became a customer of, and now he's chasing something new, agents that rewrite and retrain other agents without a human in the loop.

We dig into Sia, the meta-agent at the center of Hexo's research, and why improving an agent means touching both its harness and its actual model weights, not just one or the other. We talk about proxy evals for when you don't have much to ground truth. The Darwin-Gödel machine and why formal verification is too strict a bar for anything commercial.

How Hexo's work echoes DeepMind's Alpha lineage from AlphaGo to AlphaEvolve, and the spectrum from clearly verifiable to totally subjective tasks? Why VAE evals are quietly wrecking agent quality across the industry, and a great story about an agent that discovered a customer's own eval file was silently corrupted, something buried in hundreds of thousands of traces that no human would have caught.