CI/CD with Robert Erez

Rob Erez shares insights from his extensive experience in CI/CD, starting from his early days at Skype where weekly deployments and canary testing with New Zealand as a canary region were common. He discusses the progression to modern practices, emphasizing the importance of minimizing risks and using feature toggles to control deployments. Rob highlights that most teams aim to ship software confidently, often by rolling features forward rather than back, especially when dealing with database schema changes.

Rob explains why Kubernetes has become dominant in cloud and on-prem environments due to its declarative configuration, portability, and control. He presents real-world examples of on-prem Kubernetes deployments in various industries, including financial institutions and even on research vessels. The discussion touches on how Kubernetes interacts with infrastructure tools like Terraform and its use outside of cloud providers, emphasizing its role in enabling organizations to manage complex, multi-environment deployments.

The conversation explores GitOps, its principles, and how it emphasizes declaring infrastructure state in Git, which is then automatically reconciled with actual systems. Rob points out misconceptions, such as the idea that GitOps is solely about Git repositories, and clarifies that its pillars include declarative configuration, immutability, pull-based approaches, and continuous reconciliation. The discussion highlights that many teams prefer pragmatic solutions over dogmatic adherence to Git, especially for secrets and imperatives.

Rob discusses the emergence of platform teams as a way to manage the complexity of deployment processes in large organizations. These teams define standards, provide self-service tools, and offload operational burdens from application teams. The trend is driven by the need to scale DevOps practices without fragmenting deployment strategies or creating operational silos, aiming to strike a balance between control and agility.

Rob addresses how AI is transforming software development and deployment by increasing code output and testing automation. He points out that faster pipelines may become less critical if AI handles code generation and validation, but risk management—particularly with AI-generated code—remains vital. The conversation covers the use of feature toggles, progressive delivery, and how AI might enable more granular, safer deployments at scale.

The episode delves into progressive delivery as the next step after continuous deployment. Techniques like Canary deployments, blue-green, and feature toggles are explained, highlighting their roles in de-risking releases. Rob emphasizes that feature toggles offer precise control and quick rollbacks, especially useful when dealing with application logic changes and schema migrations. The importance of designing for incremental releases and managing complex dependencies is underscored.

Rob emphasizes the concepts of roll-forward rather than relying on rollbacks, especially when dealing with data and schema changes. He discusses the complexities of systems with state, the limitations of traditional rollback mechanisms, and recommends fast, controlled fixes—like hotfixes—to address issues quickly. The importance of feature flags for safe decommissioning or disabling features with minimal risk is also highlighted.

The discussion covers how development environments are evolving from local setups to ephemeral, cloud-based, or feature-branch environments. Rob observes that while CDEs and preview environments promise quick feedback, their complexity increases with multi-service architectures and stateful systems. The trend aims to speed up iteration, integration, and collaboration by providing on-demand, isolated environments for development and testing.

Rob reflects on how many organizations, especially regulated ones like banks, prefer or need to maintain on-prem and self-hosted CI/CD solutions due to control, compliance, or legacy reasons. He notes that these systems tend to update slowly—sometimes years behind—yet remain critical for these clients. The ongoing support for such environments influences the design and features of tools like Octopus Deploy.

Rob advises engineers interested in adopting progressive delivery to start small—such as using feature toggles—before gradually expanding. He stresses the importance of hygiene, monitoring, and fast fixes. Key lessons include the utility of feature flags for safer, more controlled releases, and understanding that most teams value reliability over raw speed. Rob also recommends books like 'The Phoenix Project' for process understanding and emphasizes that pragmatic, flexible approaches often outperform dogmatic methodologies.