i have just gotten off a productive call with sauron where i laid out our requests
- nazgul bodycams
- morgul knife must remain sheathed unless suspect is determined to be carrying the one ring
- shelob will be the new point of contact
Excellent analysis from @vanlightly.bsky.social: A Fork in the Road: Deciding #ApacheKafka’s Diskless Future jack-vanlightly.com/blog/2025/10...
New post: why I’m not a fan of “zero-copy” Iceberg tables for Apache Kafka.
From a systems design view, it trades storage savings for coupling and complexity.
Sometimes, duplication is cheaper than coupling.
jack-vanlightly.com/blog/2025/10...
Not specifically development related, but CleanShot (cleanshot.com) is so much better than the built-in screenshot app. It has scrolling capture, animated gif support, and great annotations. Also, for youtube videos, I can't live without Scribe for transcripts (addons.mozilla.org/en-US/firefo...)
Cloudflare is doing a lot of interesting work (R2 object store, cloudflare workers, pay-per-crawl for LLMs). They annouced Cap'n Web, a new RPC system that works on all JS runtimes. One of its biggest innovations over REST/GraphQL is "promise pipelining" blog.cloudflare.com/capnweb-java...
> The people succeeding with AI aren’t the ones who suddenly discovered taste. They’re the ones who already had it and simply adapted their standards to a new tool. Develop your taste with or without AI. The medium doesn’t matter, the fundamentals do.
matthewsanabria.dev/posts/you-ha...
good post on Kakfa + Iceberg from the WarpStream folks: www.warpstream.com/blog/the-cas...
Most implementations gloss over the difficulties in iceberg table maintenance, and the only solution that the marketplace seems to have is Spark. Great if your team has Spark already, but painful otherwise.
Love this Blindsight metaphor for directly sharing AI-slop output to others
Submitted a talk to Current '25 New Orleans (in October): "The Rise of Object Store Architectures: Understanding SlateDB"
SlateDB is an object-store backed LSM tree storage engine and is a building block for cloud native architectures. It's a useful lens for understanding the architectural shift.
"We all used to believe that changing passwords often was a good idea; turns out the opposite is true. Similarly, we all used to believe that making people log in frequently was good security. ... Except, well, that’s not how anything works."
tailscale.com/blog/frequen...
How to reliably distribute work across microservices, stream processors, durable execution, event-driven, orchestration and now AI agents?
Coordinated Progress is a 4 part series that explores the common structure behind reliable distributed systems.
jack-vanlightly.com/blog/2025/6/...
An increasingly common conflict I am hearing:
"Our company evaluated using {advanced AI coding product}. But we found it too expensive compared to GitHub Copilot.
So we are staying on Copilot and our platform team is building our own custom AI tools on top of open source"
Though I'll admit that I've done far less Rust programming than go/python (the two languages that LLMs feel like they're the best at because of the strong linting/formatting cultures and relatively simple syntax)
Seconding Claude Code. If you launch it inside the terminal of a VSCode derived IDE (VSCode, Cursor, Windsurf), it automatically adds a plugin that makes Claude Code aware of currently highlited code to pull into context. I personally pair it with Windsurf and like the combo.
If you're looking to execute on RaspberyPis, and can ssh to them, you wouldn't need the Kubernetes plugin, just Remote Development.
Atuin Desktop looks interesting. You mention wanting a remote version. I've used Jupyter Notebooks with a bash kernel in VSCode. With the Remote Development + Kubernetes plugins the cells are executed on a remote k8s instance. The notebooks can be checked into git and shared with my team.
Creating Apache Iceberg tables from a Kafka topic (similar to Tableflow) is another similar use case.
One thing that I'd miss with no partitions and only key-centric access is the ability to replay from a specific time/offset. My teams often create rocksdb materialized views from compacted topics. If I restore a rocksdb snapshot, I want to start consumption from the rocksdb snapshots last insert.
Got it, that makes sense. Thanks for the clarification.
The only things remotely related to that I've seen are KIP-47 (cwiki.apache.org/confluence/d... which is really old and still open) to allow for event deletion based on event time. And KIP-280 (cwiki.apache.org/confluence/d...) for compaction
Does Kafka's retention policy accomplish the same goal, or is there something additional that the low/high watermark gives?
If I've got an uncompacted topic and its `retention.ms` (kafka.apache.org/documentatio...) is set to 24 hours, segments older than 24 hours can be dropped.
also, this only works when `cleanup.policy=delete` and does not work for `cleanup.policy=compact`, or `compact,delete`. The only way to get a fully accurate count there is to iterate over everything
depending on your definition of "one single CLI invocation"
paste <(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list "<broker>" --topic "<topic>" --time -1) \
<(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list "<broker>" --topic "<topic>" --time -2) \
| awk -F'[:\t]' '{ sum += ($3 - $6) } END {printf "%.0f\n", sum}'
