We looked at Dbt originally, but our transformations and Data mart creation had so much application level logic to perform transforms that at best we'd be doing half half. What we'd use dbt for fell in the easy part of our problems.
This is one of the big enabler, with all the tools existing we could focus on domain specific problems, and with supervisors worrying about infra stability we could lean in deeper into using errors for data correctness. We plan to be very open on the content, so blogs will contain a lot of detail
I hope to publish some blogs related to this in the coming months. Originally we wanted to use as much off the shelf technologies as possible, however zappis requirements fit in a niche that made rewriting a pipeline from scratch required. In the Elixir ecosystem all the tools you need exist
I'm incredibly excited to be speaking at Elixir Conf EU this year. I'll be talking about our experience adopting Elixir and how it accelerated building our own LakeHouse Architecture.
www.elixirconf.eu/talks/do-or-...
I hope to see you there!
#ElixirLang #elixirlang #elixirconfeu
Sounds super interesting! I can't wait to see it
I'm currently trying to put together where those things were and hopefully publish something that will help the next person trying to understand those components
Personally the things that tripped me up the most were things that are obvious once you understand it. It's been primarily around how clusters, message passing and process groups work as you scale. Often I needed to dig into Erlangs docs to get the answers.
And the idea of trying to contrast that with others workflows is super exciting. I'm sure there's a lot of tricks and improvements I'll find even though objectively how it works for me is productive and progress is transparent
Small example, I agree with the sentiment, but I work with my interpretation of trunk based dev, where each commit to main is the most isolated version of the smallest unit of work, but I leave my branch with dirty commits to track progress and fiddle. Squashing makes this easy without other niggles
That's definitely a motivation to get tickets! Would love a discussion around this, I think it's super nuanced and everyone's workflow can influence the correct approach!
