Cloudflare's Billing Engine Stalls: Hidden ClickHouse Bottleneck Discovered and Patched
Billing Aggregation Slowed After Migration
Cloudflare's daily billing aggregation jobs, which process hundreds of millions of dollars in usage revenue, suddenly slowed following a migration of its ClickHouse database infrastructure. The delays threatened to cause reconciliation nightmares for invoices, according to company engineers.
All typical performance indicators — I/O, memory, rows scanned, parts read — appeared normal. The slowdown stemmed from a hidden bottleneck deep within ClickHouse's internal query execution, not from external resource constraints.
Three Patches Rescued the Pipeline
Cloudflare's engineering team identified the root cause and deployed three targeted patches to restore performance. The fixes addressed a previously undocumented behavior in ClickHouse's aggregation engine that emerged under heavy concurrent workloads.
“We had to dig into the source code and add instrumentation to find what wasn't being reported,” a Cloudflare engineer told reporters. “The bottleneck was invisible to our standard monitoring tools.”
Background: A Petabyte-Scale Analytics Platform
Cloudflare relies heavily on ClickHouse, an open-source OLAP database, to store over a hundred petabytes of data across dozens of clusters. In early 2022, it launched Ready-Analytics, a system that simplifies onboarding by allowing teams to stream data into a single massive table rather than designing custom schemas.
Datasets are distinguished by a namespace and share a standard schema with fields like floats, strings, a timestamp, and an indexID. The primary key—(namespace, indexID, timestamp)—enables efficient sorting per namespace, which is critical for query speed.
By December 2024, Ready-Analytics had grown to over 2 PiB of data, ingesting millions of rows per second. However, its original retention policy proved limiting: a rigid 31-day partition drop applied to every namespace.
The Problem: One-Size-Fits-All Retention
Because Cloudflare had been using ClickHouse before native TTL features existed, it built a custom retention system that dropped daily partitions older than 31 days. This prevented teams with longer retention needs — such as legal or contractual obligations — from using Ready-Analytics.
“The per-namespace retention flexibility was the missing piece,” explained a Cloudflare product manager. “Without it, many teams had to fall back to a far more complex onboarding process.”
What This Means
The hidden bottleneck underscores the complexity of maintaining massive, multi-tenant analytics platforms. The three patches not only restored billing pipeline speed but also improved ClickHouse's behavior under similar future conditions.
Cloudflare has since implemented a new per-namespace retention system, allowing teams to set custom data lifetimes. This opens Ready-Analytics to use cases previously excluded, reducing friction for internal customers and strengthening Cloudflare's ability to scale its billing and fraud detection systems.
Additional Resources
Related Articles
- Grafana Assistant Now Pre-Maps Your Infrastructure for Zero-Delay Troubleshooting
- How to Access Coursera's Learning Agent Inside Microsoft 365 Copilot: A Step-by-Step Guide
- Coursera and Udemy Complete Merger, But Platforms to Stay Separate for Now
- How a 1973 Book of BASIC Games Launched the Personal Computer Revolution – And Why It Still Matters
- Building a Resilient Network: A Practical Guide to Health-Mediated Configuration Deployments
- Breaking: Global GenAI Gender Gap Narrows as Women's Enrollment Surges 4 Points in One Year – Coursera Report
- Mastering ByteBuffer to Byte Array Conversions in Java
- Stop Abandoning Long YouTube Videos: A Proven Active Watching Method