Red-Teaming after Mythos — Zico Kolter & Matt Fredrikson, Gray Swan

The podcast begins with hosts Swyx interviewing Zico Kolter and Matt Fredrikson of Gray Swan, a company focused on AI security. They discuss Gray Swan's mission to enable the safe and secure deployment of AI, emphasizing the understanding and mitigation of vulnerabilities in large language models (LLMs). The founders’ background is rooted in academic research at Carnegie Mellon, exploring attack surfaces, vulnerabilities, and robustness of deep learning systems.

The conversation highlights how AI systems differ fundamentally from traditional software in terms of security. AI can be tricked in ways humans aren't, creating new vulnerabilities. Unlike conventional software, scaling up models doesn't inherently make them safer; explicit training on safety and robustness is necessary. The importance of adopting a different security mindset specific to AI is stressed, especially when AI models are used as autonomous agents integrated into larger platforms, creating complex attack surfaces.

Gray Swan operates a community red team called the Arena, engaging thousands of hackers worldwide in finding vulnerabilities in AI models via prize contests. They also develop automated red teaming models, such as Shade, capable of outperforming humans at breaking AI systems, especially in discovering indirect prompt injections and jailbreaks. The discussion covers how current frontier models resist jailbreaks due to safety training, but specialized models trained explicitly for red teaming can identify exploits more effectively.

Zico and Matt discuss AI's nature as an alien form of intelligence, quite different from human cognition. Despite debates over consciousness, they agree AI is a different, potentially more capable but fundamentally different intelligence. The scale and capabilities of models like LLMs and model interpretability research are touched upon, emphasizing that bulk increases in size don’t necessarily lead to better safety or robustness.

The potential of AI to revolutionize the scientific process, especially in interpretability and understanding model behavior, is explored. The current state of mechanistic interpretability is seen as immature, but advances in automating research—like probing activation traces and circuit analysis—could accelerate scientific understanding. There’s optimism that agents can now perform complex interpretability tasks, lowering the barrier for scientists and security researchers.

The podcast discusses practical aspects of deploying AI securely in enterprises, including the importance of guardrails like Cygnal and the challenges of managing agent permissions and identity. They explain that bigger models don’t inherently become safer, reinforcing the need for specialized guardrails and policies to prevent prompt injections and data exfiltration. They also note that organizations often start with rudimentary controls but need more sophisticated defense as they scale.

Discussion shifts to market trends like AI insurance, compliance standards, and the notion of Gray Swan events—unexpected but foreseeable security failures. The hosts highlight that major prompt injection breaches, when publicly reported, may push industry standards forward. They emphasize the importance of building a proactive security posture and potentially integrating security guardrails into the emerging AI insurance ecosystem.

Looking ahead, Zico and Matt see widespread adoption of security tools like Shade and Cygnal across industries. They anticipate that AI security measures will become a core part of enterprise AI deployment, with a focus on automating scientific research in AI safety, interpretability, and robustness. Emphasis is placed on increasing collaboration, developing industry frameworks, and automating complex security tasks to reduce reliance on human expertise, ultimately seeking to better understand and control AI systems.