Managing Mobile Apps at Internet Scale with Peter Morelli

Peter Morelli says he spent eight years as Lyft’s VP of Engineering, working with Matt Klein and Martin on the problem of understanding customer experience through mobile phones. Lyft handled 99% of its business through mobile apps and had scaled to millions or tens of millions of devices per day, but its server observability did not reveal what was happening on those phones. After roughly six years developing a different approach inside Lyft, the founders decided the technology should not remain internal: the industry still had not solved mobile observability, so they worked with Lyft’s board and CEO to spin it out as bitdrift about three years before the interview.

Morelli explains that server observability assumes a relatively homogeneous environment: thousands of similar Kubernetes nodes or VMs, where failures often cluster around a bad switch, port, or release. Mobile is heterogeneous across operating systems, hardware, carriers, radio changes, app versions, and network conditions; a driver can move between cell towers, Wi-Fi, and 5G during a session. Instead of a short 80/20 tail, mobile has a long, thick tail of device-specific edge cases. The scale also reaches millions to billions of devices, while unreliable connectivity means a phone may be unable to report a failure at the moment it happens.

Traditional observability vendors often charge by data ingestion, so companies sample aggressively—sometimes retaining only 1% or less of their telemetry. Morelli argues that this produces a misleading picture in mobile: the common, healthy conditions are represented, while the long tail of severe customer problems disappears. At Lyft, observability mattered directly to retention and growth because riders could switch to Uber with little cost. Engineers needed to understand reliability, performance, onboarding, experiments, and design changes across the phone experience, not merely whether a server received a request or a ride was eventually completed.

Lyft encountered incidents that were expensive because problems outside the company’s control could not be seen until too late. Existing commercial and open-source tools were either uneconomical at full mobile scale or depended on sending telemetry to a central location despite unreliable networks. Mobile release cycles made the delay worse: code changes could take five to ten days to pass through alpha, beta, app-store approval, and propagation, and several rounds might be needed to diagnose one issue. Morelli says the design goals became complete coverage without sampling and runtime control over what to inspect, rather than decisions fixed at compile time or the next release.

Bitdrift’s SDK collects telemetry on the device and keeps it there by default instead of immediately ingesting everything. Each application receives a bounded, compressed ring buffer—often about five megabytes for consumer apps, though point-of-sale customers may use hundreds of megabytes—so old data is overwritten without allowing the SDK to consume unlimited storage. Collection and processing run in the background, with roughly 60 milliseconds identified as the point at which users notice lag or freezes. When connectivity returns, the SDK maintains a connection to the control plane and can retrieve data from matching devices.

Rather than ingesting all data and searching it later, engineers can send conditions to the fleet: for example, find iPhones in San Francisco running particular app and iOS versions, belonging to an experiment group, and seeing an error at least four times. The device evaluates those rules and reports only matching evidence. Bitdrift also performs counting, histograms, and other processing at the edge, allowing unsampled fleet-wide metrics without transferring every underlying log. Morelli says this can expose every customer journey or guardrail metric across millions of devices while preserving the option to pull detailed logs for a specific investigation.

Morelli describes a customer that used Datadog and other traditional tools and discovered, after deploying bitdrift, that analytics and observability consumed 67% of the bandwidth on its devices—bandwidth needed for transactional processing. Because bitdrift stores data locally and computes as much as possible on the device, the customer reduced that usage by more than two-thirds. The team could also launch three or four investigations in about an hour, instead of waiting weeks or months to add instrumentation, release an app, and gather enough data. The approach is designed to work on both high-end iPhones and low-end Android devices.

Morelli says Blobstream, created by Matt Klein and open sourced, emerged because bitdrift needed to manage streams arriving from a billion deployed devices and potentially hundreds of millions of simultaneous live streams. Replacing parts of the traditional Kafka-based approach saved tens of thousands of dollars on AWS while improving performance and reliability. The SDK represents six years of hardening across tens of millions of devices, including persistence through crashes and periods of disuse. Bitdrift also partnered with ClickHouse, which already powers its metric store and offers ClickStack, so customers can combine bitdrift’s edge-focused mobile telemetry with a broader data platform.

Bitdrift exposes a public API, CLI, AI skills, and a remote MCP so agents can investigate mobile problems in cloud sandboxes or long-running environments without waiting for a new app release. Morelli says agents can build several hypotheses—network, design, or feature-flag failures—query evidence in parallel, and potentially generate a pull request. He cites ThredUp, an online consignment company, whose engineers initially reached only 20–30% of customer issues; after using bitdrift with AI-assisted investigations, they reported reaching roughly 80–90%. Morelli cautions that product design and human interaction still require human judgment, especially where friction may be intentional for fraud prevention.

Morelli argues that observability teams often underestimate the cost of storing and ingesting telemetry that no human, query, alert, or agent ever examines—he estimates roughly 90% may go unused. He dislikes quota and overage disputes triggered by events such as an AWS brownout, when customers need their observability vendor’s help rather than a larger bill. Bitdrift prices by active device instead of ingestion, aligning its revenue with customer growth and making efficiency the vendor’s responsibility. He expects the old ingestion-based model to weaken as software, AI-generated complexity, and edge devices multiply, because production behavior remains impossible to predict fully in testing.