You're probably right to pick a modern extension to support vector similarity search in Postgres. But did you know Postgres already has one built in?
I took a look at the cube extension in Postgres, pgvector, and model2vec for some impressively fast embeddings generation.
Just felt the earthquake in sf, got a short little ride on my couch 🎢
Finished the overhaul of kernel-development-flake ✨ inspired by virtme-ng which didn’t support NixOS. kdf is also a rust-based /init using virtiofs. You can now do fast live dev of kernel modules/in-tree, no more waiting for nix rebuilds github.com/jordanisaacs...
TIL rust's spawn implementation uses a unix socket for communication between child & parent (github.com/rust-lang/ru...). Only happens when it can't use posix spawn (e.g. using pre_exec callback). Hit this when running a kernel without unix sockets in qemu and got function not implemented 🫠
PostgreSQL 14 internals book
Entering my Postgres era 🐘
Well looks like they didn’t read up on the history of using io-uring safely in rust. They have the main io-uring issue of sharing buffers with the kernel. github.com/Azure/kimoji...
The classic from without boats (2020) without.boats/blog/io-uring/
Omg they are using my rustix-uring library, looks like I got to get back on the maintenance grind 😅 github.com/jordanisaacs...
And page servers are sharded (one copy stored on object storage) allowing for easy scaling up/down of page servers. The standard `GetPage @ LSN` API.
They have automatic failovers of primary to read replicas (and if old primary comes back online it will come back as a read replica). Since compute nodes are stateless this is easy for them to do. A mention of ringfencing but no details on it. Video has a nice animation of this.
HorizonDB write path diagram.
WAL writes are majority quorum from the primary. They are using rust & thread per-core io-uring on nvme. A new open source runtime, kimojio! github.com/Azure/kimoji.... Excited to dig into that.
HorizonDB high level architecture diagram.
Some interesting stuff in the talk. It’s neon style durable WAL log & async WAL replay to create pages. All backed by object storage. There is SSD cache on all nodes.
Adam prout gave a talk on it at ignite today. No recording yet ignite.microsoft.com/en-US/sessio...
Today's Future Data Systems Seminar Speaker: Joyo Victor will present @singlestore.com's "Bottle Service" meta-data system that supports database branching, change-data-capture, and Apache Iceberg. Zoom talk open to public at 4:30pm ET. YouTube video available after: db.cs.cmu.edu/events/futur...
So after a day of work I had my 3 line fix :) Add 'SHF_GNU_RETAIN' flag to the inline asm for the main flag list. And a dummy variable in the macro that uses the view sections to force a GC reference:
`__attribute__((retain,used,section("view-section")))
static const char dummy[0];`
Of course this is where I ended up at maskray’s blog doing deep reads of linker GC (maskray.me/blog/2021-02... ) and metadata sections (maskray.me/blog/2021-01...). I realized the main flag list should always exist and the view should only be kept if in use.
There has actually been a ton of consideration to handling these sort of “meta” sections that have no direct access and thus do not participate in general GC references. But upon further looking the dynamic flag didn’t use any of the special flags. So I found my bug, the linker was GCing it
Luckily I had two build flavors with one passing and the other failing. I manually went through every linker flag difference to guess which one was causing issues. And '—gc-sections' looked suspicious. The linker has some crazy complexity to how it decides to GC sections.
So dynamic flag automatically generates sections using inline assembly. One section has the list of all flags. Then there are arbitrarily many other sections that are a “view” into that list of flags for more efficient lookups. There is no access of any sections beyond using '__start' and '__stop'
But the sections existed in the object files leaving me flummoxed. Why wasn’t the linker using the sections? So I went down one of the most interesting systems rabbit holes I’ve had recently.
Been working on using github.com/backtrace-la... which does cross modifying code for negligible overhead flags. But when I tried to use it I got dreaded linker errors with the '__start' and '__stop' symbols for the custom sections that were supposed to be generated at link time missing...
Google blacklisted the iCloud private relay ip addresses. Been broken for ages while now. On iOS safari you need to press the “show ip address” option and then try searching again and it works.
Everyone complains about fighting the compiler, but the true boss battle is the linker. I’d rather have a thousand line c++ template error than an utterly unhelpful 5 line linker error.
This is really cool, TUM has a course on transaction systems theory (taught by Thomas Neumann), with open access slides in English. Stumbled upon it when one of the slides showed up in a search for commutative serializability theory.
db.in.tum.de/teaching/ss2...
I wrote up on bounding retries of write-write conflicts in MVCC+2PL. There is surprisingly little publicly written on how databases can retry, so doing my part to fill that gap in :) snowytrees.dev/blog/boundin...
Experimental off-cpu profiling landed for otel ebpf profiler 👀. It also was a blocker for using its stack unwinding as a library. Hopefully that means user space stacktraces and profiling will come soon(ish) to inspektor gadget for interactive perf analysis on kubernetes.
github.com/open-telemet...
The motivation they cite in the kernel docs is gpu buffer locking www.kernel.org/doc/html/v6....
Was searching around for open source uses of wound wait/wait die and surprisingly found it is used in the Linux kernel github.com/torvalds/lin...
Yep agreed. I initially mistook synchronous_commit it for a general async durability flag.
