What turned out to be a subtle interaction between how structs are laid out and how we serialized them also had a surprisingly simple fix.
We wrote up the full story here: www.feldera.com/blog/nobody-...
What happens when a SQL table has 700+ nullable columns?
At first glance: nothing unusual.
But when that table turns into a Rust struct with hundreds of optional fields, something odd happens. The data looks small in memory but it yet suddenly takes twice the space on disk.
TopN and BottomN Garbage Collection: bounded state from Min, Max, top-k, and bottom-k queries. Your pipelines run forever without growing.
♻️ TopN and BottomN Garbage Collection: Bounded state for Min, Max, top-k queries, and bottom-k queries.. Your pipelines run forever without growing.
Read the full product update here: www.linkedin.com/pulse/februa...
Column-level storage optimization: unused columns get skipped. Nullable columns get compressed. Wide, sparse tables just got cheaper to run.
Concurrent output encoding: Avro encoding is now multi-threaded. Output is no longer your bottleneck.
🗜️ Column-Level Storage Optimization: Unused columns get skipped. Nullable columns get compressed. Wide, sparse tables just got cheaper to run.
⚡ Concurrent Output Encoding: Avro encoding is now multi-threaded. Output is no longer your bottleneck.
Image of Feldera logo and text: "February 2026 what's new"
Star join operator: multi-way joins in a single pass. Faster star-schema queries with less storage.
This month we were busy shipping 157 changes to production including new features and improvements that make your pipelines faster, smarter, and leaner.
Highlights in this edition include:
⭐ Star Join Operator: Multi-way joins in a single pass. Faster star-schema queries with less storage.
Ingestion step size can now scale automatically with your pipeline’s worker count.
A standard 8-worker pipeline now ingests 8x more records per step than a single worker, and we measured 2x throughput improvements on some customer pipelines.
How much throughput are you leaving on the table?
As Feldera has matured, so has the engine underneath it. Last year, to support large backfills, the engine began automatically split/accumulated large outputs using our storage layer. We realized recently that this unlocked something exciting.
It's time to build AI-era products that were impossible before.
Batch processing recalculates everything, even when 99.9% of your data didn’t change.
Feldera fixes that w/ incremental compute.
Bring your existing SQL and get millisecond freshness instead of hours-long (or days-long) batch jobs.
That’s actual incremental computation.
Read more: www.feldera.com/blog/can-you...
P.S. If your incremental engine can do this, we’d love to see the results.
We ran it on Feldera:
- 200ms update latency on input changes
- Single machine (16 CPU cores)
- 15GB RAM steady state
How?
Feldera does work proportional to your changes, not your table size. When you update one or many rows, we recompute only answers that changed and nothing else.
Query plan for a complex, real-world SQL program
This is a query plan for a real customer’s production SQL.
- 61 input tables -> 33 output views
- 217 joins
- 27 aggregations
- 218 projections and filters
Most systems would either do an expensive full recomputation or flat out fail.
Your pipeline was disrupted.
Feldera’s Health Page tells you why in seconds. No K8s access. No waiting on DevOps.
Details -> www.feldera.com/blog/introdu...
Introducing Feldera Health 🩺
A lightweight health monitoring solution built directly into Feldera. See the real-time status of your compiler, API server, and runner at a glance.
✅ Available today on try.feldera.com and Enterprise Feldera
📝 Detailed technical blog coming soon
Incremental Updates - January 2026 edition is here! 🚀
This edition covers:
- Product updates: adaptive join rebalancing, GC for ASOF joins and more
- New blogs: deep dive into our profiler, constant folding in Calcite, and a look back at our progress in 2025
www.linkedin.com/pulse/januar...
🩻 X-ray vision for your SQL pipeline in Feldera.
-Click any node -> see metrics across all cores.
-Heat map shows bottlenecks instantly.
-Expand to trace back to your SQL code.
⚡ Seconds to see what used to take hours to find.
Dive deeper: www.feldera.com/blog/introdu...
2025:
📦 166 unique releases, 1,162 changes, avg. new release every 2.4 days
📊 10x cost reduction for users, hours old insights into sub-second latency
⚡ 70-node Spark clusters -> single digit Feldera instances
2026: Make incremental compute inevitable
Full story: www.feldera.com/blog/feldera...
Because when you're processing millions of changes per second, you need to see what's happening to optimize it.
Read the technical deep-dive: www.feldera.com/blog/introdu...
📊 Real-time metrics per operator: execution time, memory, data volumes, disk I/O
🎨 Visual dataflow graphs with color-coded performance heatmaps
🔍 Interactive exploration - click any operator to see detailed breakdowns
That’s why we built the Feldera profiler. It shows exactly where your computation time and resources are going.
You can’t optimize what you can’t profile.
Which operators are a bottleneck? Are there skewed joins? Why is storage use spiking?
Our engineering team used to spend hours trying to answer these questions when performance problems would show up in the wild.
They’re all available today. Now go build something fast. 🏎️
Backfill avoidance: modifying a query used to mean having to recompute & backfill all over again. With backfill avoidance, you can avoid another backfill by reusing existing states when applicable, & recomputing only what’s new. Much faster than starting over.
docs.feldera.com/pipelines/mo...
Fast backfill (HUGE steps): you know backfilling historical data can take forever. That's why we shipped a transaction API where you control the batch size - whether small or HUGE - for efficient bulk ingests.
docs.feldera.com/pipelines/tr...
Parallel compilation: we used to compile pipelines one at a time. Now we distribute the workload across multiple servers. Pipeline builds that were queued back-to-back can now complete simultaneously within minutes. It shaved an hour off of our CI pipeline!
docs.feldera.com/get-started/...
Rust Compilation can be slow. Backfills can take days. And no one likes to wait. ⏳
We’ve recently shipped features that will get you deploying faster and scaling more efficiently:
- Parallel Compilation
- Fast Backfill
- Backfill Avoidance
If you’re excited about hard technical problems and want to shape the future of real-time systems, Feldera is hiring (remotely!) for a Solutions Engineer (Enterprise) and a Software Engineer (Reliability, Performance)
jobs.ashbyhq.com/feldera/544a...
Hi Ben, I wanted to follow up on my recent note about your containerized solutions. If Feldera is exploring ways to improve mobility or streamline deployment, our container wheels could be a great fit. Here’s what our solutions offer: Increased flexibility Reduced operating costs Safe and practical mobility A more sustainable option With logistics hubs in both the US and EU, our casters are already helping teams in container rental, storage, and logistics cut down on equipment costs, labor, and time.
Some of my work at @feldera.bsky.social involves containers. I keep getting spam from some vendor who wants to sell me casters to put on the containers 🤣
"This is the true power of incremental compute. By only needing compute resources proportional to the size of the change, instead of the size of the whole dataset, businesses can dramatically slash compute spend for their analytics."
👇
www.feldera.com/blog/how-fel...
At @feldera.bsky.social I've been doing a lot of performance work. I needed an easy way to watch the Prometheus metrics for a pipeline, so I wrote a simple tool for the Feldera CLI that shows the pipeline metrics. Here's the progress of a pipeline that runs in about 30 seconds.
