EP264 Measuring Your (Agentic) SOC: Two Security Leaders Walk into a Podcast

The discussion begins with an analysis of traditional SOC metrics like Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). The speakers argue that these metrics are becoming less relevant as AI handles incident triage and resolution at machine speed. Instead of focusing solely on metrics like speed, the conversation shifts toward the importance of metrics that measure quality and context, such as 'Time to Context,' which accounts for understanding and action rather than just detection speed. The speakers emphasize that metrics need to evolve with technological advancements, highlighting that some traditional indicators could even be considered 'vanity metrics' if they no longer reflect true security health.

The podcast explores the 'Maturity Triangle' framework, which assesses security maturity across infrastructure, operating system, applications, and data layers. The integration of AI is expected to shift this triangle, possibly moving organizations from a 'people-heavy' model to a more 'engineering-led' approach. The discussion highlights that as automation and AI expand, the metrics tracking the effectiveness of detection, response, and automation need to adapt. How organizations measure success, such as toil reduction, is also debated, with an emphasis on metrics that show the actual impact of AI—like time saved or threat complexity reduced—rather than just speed.

Google's internal security strategy prioritizes quantifying 'Toil Reduction'—the reduction of repetitive manual tasks—by tracking how many incidents are handled automatically versus manually. The metrics also include the quality of incident handling, such as the accuracy of AI-assisted triage and the time saved for security analysts. Google measures success by assessing operational efficiencies, risk reduction, and the risk exposure within the organization's appetite. For the board, high-level metrics focus on risk mitigation and incident reduction rather than raw detection counts, emphasizing 'risk exposure' within a company's risk appetite.

The panel discusses the economic implications of integrating AI into security operations. AI is viewed more as an augmentation tool that increases analyst productivity and decision accuracy, rather than a direct replacement for humans. Metrics such as the ratio of AI-handled to human-handled incidents are considered crucial, but there is caution about over-reliance on automation, which could introduce new risks if AI handles instances improperly. The conversation underlines that as AI advances, organizations will need to redefine success metrics—moving toward measures that reflect risk mitigation, efficiency gains, and operational maturity—while understanding that the economic landscape for security staffing and automation is constantly changing.

The speakers acknowledge the challenges posed by regulations and audits when deploying non-deterministic AI systems in security. Explaining and justifying AI decisions to auditors and regulators is complex, especially regarding false negatives or missed detections. Trust and transparency are central themes, with a focus on maintaining auditable processes, comprehensive documentation, and proving risk reduction. Google emphasizes that while AI can assist significantly, human oversight remains critical, especially in regulated environments, and success will depend on how well organizations can communicate AI's role and effectiveness without relying solely on deterministic measures.

The discussion on high-level metrics reveals that organizational and board metrics tend to focus on risk reduction and overall security posture rather than detailed operational data. For example, rather than sharing pipeline latency or detection speed, security leaders communicate in terms of risk exposure and incident impact, demonstrating how security investments reduce their overall risk within acceptable thresholds. This approach aligns security metrics with business and financial outcomes, making them more relatable and meaningful for executive decision-makers.

The conversation concludes with insights into the economic and workforce implications of AI in security. AI is seen as a productivity enhancer—helping security teams do more with less and shifting their roles toward more strategic, complex tasks. Analysts' jobs are evolving rather than disappearing, focusing on deep investigation and decision-making. While automation may reduce some manual toil, the importance of human judgment remains. The panel emphasizes that success in the future will depend on organizations' ability to adapt their operational models, redefine what constitutes value, and measure improvements in risk, efficiency, and threat understanding.