WarpStream

What did Robinhood gain by switching from Kafka to WarpStream for its logging workloads?

💸 A 45% cost savings.
📊 Auto-scaling that keeps clusters right-sized.
👍 Simpler ops (no more noisy neighbors or complex networking).

buff.ly/4wy5WPV

Cost-Effective Logging at Scale: ShareChat’s Journey to WarpStream By switching from open-source Apache Kafka to WarpStream, ShareChat was able to implement zero-ops auto-scaling and saved 60% vs. multi-AZ Kafka. They also shared some best practices for optimizing…

By switching from open-source Apache Kafka to WarpStream, ShareChat was able to implement zero-ops auto-scaling and saved 60% vs. multi-AZ Kafka. They also shared some best practices for optimizing WarpStream.

Minimizing S3 API Costs with Distributed mmap We first introduced WarpStream in our blog post: "Kafka is Dead, Long Live Kafka", but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object…

When S3 is your storage engine, you have to minimize the number of PUTs and GETs you use, as they are the same charge regardless of file size. Learn how we optimize read + write paths to drive down costs.

Need to get Kafka topics into Iceberg tables? Don't use Spark or rely on tiered storage. Use WarpStream Tableflow. It's so simple that it almost feels like magic and does it with low latency while taking care of compaction 👉 buff.ly/Xhl5atr

WarpStream Multi-Region Clusters Overview WarpStream Multi-Region Clusters guarantee zero data loss (RPO=0) out of the box with zero additional operational overhead. They offer multi-region consensus and automatic failover handling. Learn…

WarpStream's Multi‑Region Clusters replicate every write across 3 regions, delivering true RPO=0 and a 99.999% uptime SLA. Never lose a record, even during region‑wide outages.

Blog: buff.ly/mIyzF3a
Product page: buff.ly/zKId6zg
Docs: buff.ly/JmJfL89

Audit Logs for WarpStream: Full Visibility Into Every Action on Your Clusters Audit Logs give you a complete, structured record of every authentication action, authorization decision, and platform operation across your WarpStream clusters.

"Who deleted that?" WarpStream Audit Logs give full visibility into every auth decision, every admin action, and every cluster operation. Structured, consumable via Kafka, and ready in one click.

Taking out the Trash: Garbage Collection of Object Storage at Massive Scale Distributed systems built on object storage all have one common problem: removing files that have been logically deleted either due to data expiry or compaction. We review the pros and cons of five…

🗑️ Distributed systems built on object storage share a common problem: removing files that have been logically deleted, either due to data expiry or compaction. What's the solution? We'll give a hint: it's not bucket policies or synchronous deletion. 🪣

ACLs are needed for security, but enabling them in production carries risks. WarpStream ACL Shadowing evaluates ACLs before enforcing them, so engineers can identify and fix issues safely 👉 buff.ly/6rCKhLi

Did you know Cursor's Kafka-compatible data streaming infrastructure is powered by WarpStream? Learn about:

🙌 Why they picked Warpstream over OSK
⏲️ Latency
🔒 Data privacy
📊 Scaling and zero ops
🚀 And more

buff.ly/tKYhA9J

Two features make WarpStream the fastest, most reliable diskless Kafka product ⤵️

⚡️ Lightning Topics: Ultra-low produce latency with a p50 of 33ms and p99 of 50ms.
🪂 Ripcord: Continue processing produce requests even when the Control Plane is down.

buff.ly/wpd1TIC

The Kafka Metric You're Not Using: Stop Counting Messages, Start Measuring Time Traditional offset-based monitoring can be misleading due to varying message sizes and consumption rates. To address this, you can introduce a time-based metric for a more accurate assessment of…

Measuring consumer group lag in Kafka using offsets isn't always the best approach. Time lag makes it much easier to monitor and troubleshoot consumer groups. Time lag measurement is built directly into WarpStream – no third-party tooling is needed.

Dealing with rejection (in distributed systems) Backpressure is a really simple concept. When the system is nearing overload, it should start “saying no” by slowing down or rejecting requests. Of course, the big question is: How do we know when we…

Dealing with rejection (in distributed systems):

🤔 Theory vs. practice.
📈 What is backpressure?
✨ The backpressuring Goldilocks zone.
📉 How we manage backpressure for our Produce and Fetch requests in our Agents (with before-and-after results).

WarpStream separates storage, compute, and metadata. The cloud provider manages the storage, the customer manages the stateless compute (the Agents), and WarpStream manages the metadata. No raw data ever leaves your environment ➡️ buff.ly/XImUEa5

Cloud disks are really expensive. They're often 3-5x more than teams realize once you factor in IOPS, throughput provisioning, and replication. Object storage changes the equation entirely 👉 buff.ly/khYRwAB

How WarpStream Agent Groups help with Kafka demand spikes https://www.warpstream.com/agent-groups | Viktor Gamov is at the lightboard to illustrate how WarpStream Agent Groups help Apache Kafka during unpredictable demand spikes. While traditional Kafka…

WarpStream Agent Groups isolate workloads within the same Kafka environment, enabling consistent performance, simplified operations, and seamless scalability, without sacrificing flexibility or blowing your budget. Learn more 👉 buff.ly/6hHVe5B

Robinhood's Use of WarpStream for Logging In this Current presentation from Robinhood software engineers Ethan Chen and Renan Rueda, they talk about how switching from Kafka to WarpStream for their logging workloads saved Robinhood 45%.…

What did Robinhood gain by switching from Kafka to WarpStream for its logging workloads?

💸 A 45% cost savings.
📊 Auto-scaling that keeps clusters right-sized.
👍 Simpler ops (no more noisy neighbors or complex networking).

buff.ly/TqPvWNA

Audit Logs for WarpStream: Full Visibility Into Every Action on Your Clusters Audit Logs give you a complete, structured record of every authentication action, authorization decision, and platform operation across your WarpStream clusters.

"Who deleted that?" WarpStream Audit Logs are here. Get full visibility into every auth decision, every admin action, and every cluster operation. Structured, consumable via Kafka, and ready in one click.

How Superwall Uses WarpStream and ClickHouse Cloud to Scale Subscription Monetization With WarpStream and ClickHouse Cloud, what was once a patchwork of managed services and constant disk resizing has become a cohesive architecture that connects streaming and analytics in a single,…

@superwall.bsky.social moves 100 MiB/s of events from @warpstream.com into @clickhouse.com Cloud, feeding a dataset of over 300 TB of total data that’s growing by 40 TB each month. WarpStream reduced Superwall's Kafka bill by 57%.

🍱 Bento is the free, open-source stream processing framework maintained by WarpStream 🍱

1️⃣ Blog about its origins: buff.ly/0X18NQS
2️⃣ Bento website: buff.ly/h9AScmU
3️⃣ How it powers Managed Data Pipelines: buff.ly/wmcwRRy

WarpStream Tableflow's architecture takes inspiration from React's virtual DOM, enabling exactly-once ingestion and millions of Iceberg updates per second while ensuring data lake correctness ➡️ buff.ly/f0mvMFR

Beyond Tiered Storage: Deep Dive into WarpStream's Storage Engine In this talk, we'll discuss in detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka® protocol compatible system that has zero local disks. We'll cover…

Interested in WarpStream's diskless architecture? Check out this technical talk where we review our cloud-native design and its benefits. Learn even more about our architecture 👉 buff.ly/e9Sh8cj

Orbit creates identical, inexpensive, scalable, and secure continuous replicas of Kafka clusters. Learn more ⤵️

🛰️ Blog: buff.ly/EjIaDfJ
✅ Docs: buff.ly/WsXxu7h
📡 More info: buff.ly/qaLlDj3
👨‍🏫 Overview video: buff.ly/gh0ZsIB

What is WarpStream by Confluent? (A Lightboard by Tim Berglund) https://www.warpstream.com | What is WarpStream by Confluent, and how does it differ from every other hosted Kafka service in the world? Tim Berglund (VP, Developer Relations) explains the different…

What is WarpStream? @tlberglund.bsky.social explains storage and compute resources, the trade-offs in BYOC offerings, and how WarpStream provides a more secure and cloud-native solution. Learn more about BYOC 👉 buff.ly/3PoYSVC

The Art of Being Lazy(log): Lower latency and Higher Availability With Delayed Sequencing We're excited to launch Lightning Topics and Ripcord Mode. Lightning Topics use delayed sequencing to achieve 33ms median produce latency with S3 Express One Zone — a 70% reduction. Ripcord Mode lets ...

Two new features make WarpStream the fastest, most reliable diskless Kafka product available ⤵️

⚡️ ️Lightning Topics: Ultra-low Produce latency with a p50 of 33ms and p99 of 50ms.
🪂 Ripcord Mode: Continue processing Produce requests even when the Control Plane is unavailable.

The Original Sin of Cloud Infrastructure Many of today's most highly adopted open source “big data” infrastructure projects – like Cassandra, Kafka, Hadoop, etc. – follow a common story. A large company, startup or otherwise, faces a…

The phases of cloud infrastructure:

1️⃣ Software is given away. Add-ons sold.
2️⃣ Cloud migration. Costs went up instead of down.
3️⃣ Lift and shift BYOC with lots of fees.

Learn how WarpStream was developed as cloud-first to solve these issues.

WarpStream Tableflow is now GA. Tableflow is the easiest, cheapest, and most flexible way to convert Kafka topic data into Iceberg tables with low latency, and keep them compacted 👉 buff.ly/PvaCika

Getting Rid of Kafka Noisy Neighbors We look at what noisy neighbors are, the current ways to handle them (cluster quotas and mirroring clusters), and how WarpStream's Agent Groups feature compares in terms of elasticity, operational…

Kafka's "noisy neighbors" problem (where workloads often compete for resources) can lead to increased latency, performance degradation, and failures. We review how to handle it with quotas, mirroring, and WarpStream Agent Groups ➡️ buff.ly/GdZU7h2

What are the benefits of WarpStream's zero disk or diskless architecture?

💸 Reduced costs.
📈 Auto-scaling.
🎯 Workload isolation.
📊 Easy integration with Managed Data Pipelines.
🔒 Zero access and BYOC native.

buff.ly/tCrRUXI

Pixel Federation was using Amazon MSK, and their Kafka bill was growing faster than their data volumes and they had to use complex networking like VPC peering. Learn how switching to WarpStream cut their bill and simplified operations ➡️ buff.ly/tvk4AJM

Learn how WarpStream leverages @antithesis.com to do deterministic simulation testing for our entire SaaS — from sign up to creating a virtual cluster to producing and consuming data, and why we chose it over Jepsen 👉 buff.ly/Scdr36b