Philosopher Ping

Hybrid Store Design Pattern 🔂

You can see the full post, along with all diagrams, here: philosopherping.com/hybrid-store...

Hybrid Stores are a way to implement Lambda Architectures, but by merging at write-time, rather than read-time. They can also support Kappa Architectures.

@venicedb.org is an open source database implementing the Hybrid Store design pattern, and other proprietary systems do so as well.

- After the buffer replay is caught up, switch the current version pointer from N to N+1. Version N can be kept as a backup. There is no future version anymore (until the next batch job).

- Replay recent data from the real-time buffer onto the future version N+1.

When the batch data source runs and produces a new version of the dataset, the Hybrid Store design pattern then does the following:

- Create a new version N+1 of the dataset (called a future version) and load the batch data into it.

The batch data source is a job that produces new versions of the dataset. The real-time data source, on the other hand, appends data to a real-time buffer, and this data eventually gets written to all versions of the dataset.

Hybrid Store Design Pattern

The Hybrid Store is a design pattern for merging batch and real-time data sources into a single dataset. It is an abstraction composed of several versioned datasets, along with a current version pointer (or symlink) serving one of them.

The Coupling Cost of Leverage ⛓️

Read the rest here: philosopherping.com/the-coupling...

The Coupling Cost of Leverage ⛓️

This maintenance gain could be especially significant if system B offers a correct solution to a difficult problem. That being said, some nuances are missing.

The Coupling Cost of Leverage ⛓️

The case in favor of A leveraging B was that it would decrease the maintenance cost of system A, since more functionality would be “outsourced” to system B (owned by another team).

The Coupling Cost of Leverage ⛓️

The Coupling Cost of Leverage 🧵

This is a summary of a recent discussion about making some system A leverage some other system B. The new system B would replace some functionality which system A already had.