Mer

a man in a hat is sitting in front of a lit candle .. ALT: a man in a hat is sitting in front of a lit candle ..

Omg this is perfect.

Bangarang!

No problems for me

a man giving a thumbs up in front of a sign that says diemarysues ALT: a man giving a thumbs up in front of a sign that says diemarysues

I give this feature the same reaction I give to dumb drivers.

Context switching has rendered me useless

The Power of Detail in Data Attention to Detail

I started posting on my substack again to get better at writing. I'm aiming to just post about my experiences as a data professional and how my past experiences have helped me.

Here's my most current post if any are interested.

substack.com/home/post/p-...

What did you use for your warehouse? The volume we work with is not much so I was just thinking of azure SQL database.

I think that's a great idea to showcase hopefully diverse workflows.

Yes, data scientists. However looking back now I should have asked what you meant with workflow workshops? I just assumed it meant "how they do the things they do"

Does anyone here have a data stack using resources in the Azure Marketplace? I'm looking to get a data warehouse going for my small team What do you use for:

- ingestion
- transformation
- storage
- orchestration

#databs

I'd like to learn about what problems they solve and how they choose the tools to solve them. And how those tools integrate with each other.

Resume/LinkedIn wise I'd say someone with a diverse background (industry/domain). I'd figure they would have had to solve different problems in different ways using different tools?

Curious why you want to be able to ID a generalist?

I think part of the reason I posted about my team's stack is because of the lack of representation with the older tech stacks. and People shouldn't be ashamed to talk about it even though it's not what the majority of the posts are talking about.

Data SMEs need to have a seat at the table when discussing KPIs or OKRs. Especially when those metrics determine compensation actions. If you don't, you run the risk creating a metric that can't be measured or over promising on measuring capabilities.

Wish I did live so far.

Me: “Hey, can I set up some time to go over your code that uploads a file to share point?”

Dev: “sure, I’ll need a code to charge my time to”

Does anyone else experience this?

TIL to inspect/clean my data prior to merging in pandas. Output had 0 rows. Def something with the join. Inspected the output of each df prior to the merge and saw the column I was joining on had extra spaces. Trim function worked. db2 to SQLServer data joined. 80 sec vs 2 hours in power BI

2025-02-09

TIL that BQuery uses a columnar storage format, meaning columns are stored separately. Important to know since BQ pricing is based on data processed by queries. Make sure to only bring in needed columns and LIMIT for sampling.

2025-02-02

TIL to orchestrate the same flow as 2025-02-01 except store in GCS and load to BigQuery. Going through the exercises gave me a high-level understanding of Kestra. Practice through projects will help me understand the details by creating flows from scratch. #dezoomcamp

2025-02-01

TIL to set up a Kestra to orchestrate extract & load of csvs into a separate pg container. Learned scheduling and backfilling data. Sensing that DE is configuring diff technologies together. Helps to have a higher level of understanding before diving into configuring. #dezoomcamp

Ghostty Docs Ghostty is a fast, feature-rich, and cross-platform terminal emulator that uses platform-native UI and GPU acceleration.

Yes!

This guy is awesome - www.youtube.com/watch?v=ZDV4...

Terminal emulator - ghostty.org/docs

Ghostty config/zsh plugins - www.bitdoze.com/ghostty-term...

Prompt = starship.rs/guide/

Last week instead of learning about Kestra for orchestration, I used 'productive procrastination' to learn about the terminal, shells, prompts and plugins to achieve a beautfil terminal. Proud to say I achieved both form and function on my little sidequest. Here's what it looks like now.

lol. No. “Documentation” is step 1, run python script. Step 2, open tableau prep, log in and run the flow. Step 3, refresh power BI workbook. Step 4, publish.

a young boy wearing a florida shirt is sitting in a car seat ALT: a young boy wearing a florida shirt is sitting in a car seat

#dataBS

On Fri I was handed off a dashboard to update/maintain. The analyst combined excel files using python with a csv output . Then Tableau prep to join the data from the python output to data from SQL Server. The tableau prep flow output a csv which was then used by Power BI to create the dashboard.

Great idea with the video! I’d love to be part of the community you want to build. Can I help somehow?

I think its kinda nuts the things you can do with just a few lines of code, like building a whole infrastructure.

2025-01-21

Started learning about Kestra. I'm seeing similarities between Kestra's "Flow" files, Docker's "Compose" file and Terraform's files. TIL that while they look the same, they are not as Kestra and Docker are YAML while Terraform is HCL. All are forms of Data Serialization files.

Failure is an integral part of growth and learning.