Listener Questions: Ops Edition

Kimberly Rhodes introduces the episode of the Rework podcast and welcomes Aaron Nicholson from 37signals' operations team. Aaron shares his background, role as a site reliability engineer (SRE), and his journey within the company. The discussion highlights the diverse and regional structure of the ops team, emphasizing their collaborative and cross-functional approach to managing systems and infrastructure.

The podcast delves into the organization of 37signals' ops team, which is split into regional groups (Americas, Pacific, Europe). Each team is part of the same on-call rotation, enabling coverage across different time zones and reducing burnout. The team includes specialists such as database and elasticsearch experts, and they leverage automation tools like Chef to maximize efficiency despite the small size of the team relative to the company's scale.

Aaron discusses the company's approach to on-call duty, including regional coverage to follow the sun and minimize sleep disruption. They have implemented structured schedules to prevent burnout, such as rotating responsibilities and ensuring primary on-call staff handle most issues. The team admits that while outages have occurred, proactive strategies like data center redundancy and infrastructure resilience have kept incidents minimal, with notable efforts to maintain active-active setups between Chicago and Virginia for high availability.

The conversation covers 37signals' strategies for maintaining system uptime and disaster recovery. They have established multiple data centers, including active-active configurations, and are exploring new locations like Amsterdam to improve European performance. These efforts are part of their broader cloud exit initiative, which involves complex projects like bringing critical services in-house and deploying geographically distributed read sites to enhance speed and reliability.

The team manages ongoing operational tasks alongside long-term projects by integrating them into their cycle planning. They acknowledge some projects span multiple cycles, such as migrating Basecamp 4 to a new infrastructure. Clear communication about project timelines and collaborative work across teams helps manage expectations and keeps progress steady. Aaron emphasizes transparency and the importance of setting realistic goals for large initiatives.

Aaron highlights their innovative use of networking technologies like BGP Anycast for traffic routing and their plan to extend this to new regions, including Europe and possibly Japan. These initiatives aim to improve global performance and demonstrate the company's commitment to resilient, scalable infrastructure. Aaron also encourages sharing their technological approaches via blog posts to foster community learning and transparency.