I sat down with Denis Magda to talk about Postgres.
He literally just published a book called “Just Use Postgres”.
In this conversation, that turned out to be super fun, we talked about everything from what MySQL got wrong to the recent PG dev explosion
Watch here👇
www.youtube.com/watch?v=LxQy...
Stay tuned on our YouTube Premiere tomorrow:
✅ www.youtube.com/watch?v=pJQ7...
PS: Happy New Year! 💥🎉
- low-level performance optimizations including in-depth flame graphs analysis
- the need for proxies in Kafka
- Kafka's java bias
- a lot more
I know it may sound like "yet another Kafka", but it's the one I'm genuinely most excited about. Trust me there's good reason why 🙌
The chat is full of nerd-snipes, where we talk about basically everything, including:
- the characteristics and value prop of Tansu
- our combined decade and a half experiences with Kafka
- schemas and today's complexity around them
- SQL + Streaming
- Rust for building low-level systems
You can tell he's incredibly passionate about programming and Kafka from the super-engaging conversation we had. If you’d like to hear a no-bs techincal engineering conversation - watch me tomorrow as I interview him for 3 hours and 30 minutes straight.
It's the story of a one-man-show - Peter Morgan - a super-productive solo Rust developer with the desire to build a world-class ergonomic Kafka broker UX that's optimized for developers.
Have you heard about Tansu?
Tomorrow, I will post the longest-ever podcast about it.
It’s an open-source (Apache), stateless, leaderless, single-binary Kafka broker supporting pluggable storage backends (S3, PostgreSQL, SQLite) with built-in schema registry and support for open-table formats
Arguing by quoting the raw numbers of posts taken down is very naive because in social media there's always a very small tail of high-performing posts.
Data retention in disk/memory is decoupled from whether the message was consumed or not. Unlike certain message queues, this ensures availability is unaffected (can’t run out of disk/memory) from how consumers behave
A leader/follower replication model does not require strong distributed consensus between too many nodes to serve traffic.
Partitions act as a sharding mechanism. A single partition usually does not push more than a few MB/s.
A Kafka cluster scales by horizontally adding more nodes (called brokers).
The append-only nature of the log (i.e no updates/deletes) means that reads do not require a quorum to serve fresh data.
There are also no locks between concurrent reads or writes, hence no contention.
Kafka heavily leverages the OS file system cache (page cache) to serve reads fast from memory
Physical disk writes are heavily batched on the server-side. The fact that Kafka writes linear blocks to disk and fsync is not used allows the OS to batch up data in memory (page cache), coalescing the data into larger IO operations
- since the synchronous write path is literally writing to memory. Producers can also be configured to not block on this replication consensus when writing (acks=1 vs acks=all).
Writes by default require consensus that every follower replica has acknowledged the write before returning, but Kafka works with fsync off by default and therefore writes to disk asynchronously, without blocking on the physical disk write. This provides near memory-like performance when well-tuned-
The broker does not touch the data, its a dumb pipe - messages aren’t even validated for their schema, hence compressed data isn’t decompressed on the server side
Compression is usually also done on the client, and these larger batches compress much more favorably. This leads to larger network packets, larger sequential disk operations, contiguous memory blocks
Writes are batched on the client-side. Producers batch multiple messages into one "record-batch", hence increasing throughput per request.
The system supports two main very simple and performant operations:
- Produce, which appends to the end of the log
- Consume, which reads sequentially starting from any particular offset
Why Kafka performs so well 👇
🧵
The Android feels choppy/flow and the touch also doesn't feel super responsive
github.com/tansu-io/tansu
This is something to keep an eye on. Should be doable to add within a pg extension and bundle Kafka into Postgres?
Daylight is amazing. I'm really looking forward to a v2 (whenever it comes) that's a bit more refined
I mean, yeah - it's not a technical piece, it's a financial article.
When I hear the Kafka community, I think about engineering talk more than anything else
With greater competition and commercialization, it seems bound to happen.
I'm of the opinion we're bound to see consolidation in the space soon, because there's too many companies chasing too little of a market: bigdata.2minutestreaming.com/p/event-stre...
Then I don't see why a relational database can't maintain a pending queue too. Just process the lowest-id job
What are you referring to?
My intuition is rather that discussion has somewhat died down, in general.
the data bible dropping soon
