If anyone out there self-hosts @dagster.io on GCP, I would love to connect. Looking for advice/ suggestions on on avoiding any pitfalls!

Whoa this is a really interesting acquisition. This feels like old school vs new school

I've had fairly limited responses on other platforms - hoping this crew has some thoughts!

Anyone have experience dynamically building dbt dependencies within the graph? ie. run the same pipeline with multiple params, but some runs require bespoke models that need to be swapped in for the default

For some reason this feels more difficult to reason about than a LookML ownership/ implementation but Im not sure why...

For those of you using @cubedev.bsky.social (or any semantic layer), who "owns" the models? Are you finding that data engineers are doing that, the consumers (ie. Frontend devs), or someone in the middle?

Should these models be treated as an extension of the transformation layer?

Is anyone here a part of the Data Engineer Things slack? Does it still exist?

Airflow optimizations irk me.

The fact that inline imports are an anti-pattern in all other python tooling but are an officially suggested method to improve your Airflow runtime is bonkers.

Automate The Boring Stuff. That book changed the trajectory of my career, no question

I'd like to think the most effective people in each of those roles have doing both for some time - the distinction existed more in marketing material, job postings and LinkedIn than in the real world for most.

Though I see a job posting for "data analytics engineer" the other day 🤦

My nespresso had the exact opposite effect on me haha

This sounds cool. Are they all single servings per day?

Oh yeah I meant the general hate for it out in the wild, not from you specifically!

I’ve never understood the hate for the medallion architecture. It’s just a different set of terms for some generalized patterns

TIL about the Norway Problem

ruudvanasseldonk.com/2023/01/11/t...

Yes and no. I think Mongo has historically sold the idea of providing fast time to value, which the business loves. S3 as a data store is positioned as an ease of use thing but seldom consumed directly by end users.

“lake house” sounds cooler anyway ha

Seems too good to be true but I think all just jaded from the Twitter bots

I am just now discovering the convenience of VSCode linked to GitHub Codespaces and Im slightly embarrassed that I didn't do this sooner

This whole feature set kind of terrifies me but it is pretty cool

Not there yet but that will be the eventual output.

This actually helped point me in the right direction. Took some digging but it requires a combination of fields (timestamp + log sequence number)

Anyone well versed in Google DataStream CDC?

Im looking to stream Postgres changes to BigQuery; is there a way to get the timestamp BQ becomes aware of a row when writing changes to BQ directly?

Alternative is to write to GCS then ingest to BQ, but wondering if this is possible with direct to BQ

JSON support on Redshift is not nearly as friendly.

You cant as easily size up your compute on an ad-hoc basis so complexity of the workload matters.

Are you managing the cluster at all? There are semi-easy ways to tune performance, but it is also not fun if you don't know what you're doing.