Data Code 101

Anthropic Courses Browse all Anthropic courses

Anthropic courses for free
#AI #Agents #Claude
anthropic.skilljar.com

The future of software engineering

AI is changing software engineering by shifting the focus from writing code to supervising AI agents.

The future requires new tools, practices, and roles that help humans and AI work together effectively.
www.thoughtworks.com/content/dam/...

Power BI connects to the warehouse SQL instance using a gateway or direct connection.

Builds relationships between dimension and fact tables, defines measures like Total Sales, Orders, Avg Order Value, and filters by date/restaurant/region.

Load

Insert/update into warehouse tables, usually with upsert logic for slowly changing data like restaurant or menu details.

Create indexes and possibly summary/aggregate tables to speed up BI queries.

Transform

Data quality: handle nulls, fix invalid values, standardize timestamps and currencies.

Business logic: derive status (completed/cancelled), order duration, delivery time, etc.

Dimensional modeling: create dimensions and facts with surrogate keys.

Extract

Periodic jobs (e.g., stored procedures, scripts, or an external tool) read new/changed rows from the OLTP MySQL database.

Data is loaded into staging tables without heavy logic, often as 1‑to‑1 copies of source tables plus load metadata.

Data warehouse / reporting schema

Dimensional or star‑like tables (e.g., dim_customer, dim_restaurant, dim_date, fact_orders) are built for analytics.

Staging schemas

Raw tables may be copied or materialized into staging tables where basic cleaning, type fixes, and simple joins happen.

Source OLTP DB

Tables like customers, restaurants, menu items, orders, order_items, payments hold raw, highly normalized data optimized for the ordering app, not reporting.

End-to-End Data Engineering Project: Food Order ETL Pipeline using MySQL & Power BI
#dataengineering

This project shows a full ETL/analytics flow for a food‑ordering business, from raw operational data in MySQL to interactive dashboards in Power BI.

Flow of data and where people experience the problems
Image by Matt Arderne (Forbes)

Prompting is temporary.

Structure is permanent.

When your repo is organized this way, Claude stops behaving like a chatbot…

…and starts acting like a project-native engineer.

5️⃣ Local CLAUDE.md for risky modules

Put small files near sharp edges:

src/auth/CLAUDE.md
src/persistence/CLAUDE.md
infra/CLAUDE.md

Now Claude sees the gotchas exactly when it works there.

4️⃣ docs/ = Progressive Context

Don’t bloat prompts.

Claude just needs to know where truth lives:

• architecture overview
• ADRs (engineering decisions)
• operational runbooks

3️⃣ .claude/hooks/ = Guardrails

Models forget.

Hooks don’t.

Use them for things that must be deterministic:

• run formatter after edits
• run tests on core changes
• block unsafe directories (auth, billing, migrations)

2️⃣ .claude/skills/ = Reusable Expert Modes

Stop rewriting instructions.

Turn common workflows into skills:

• code review checklist
• refactor playbook
• release procedure
• debugging flow

Result:
Consistency across sessions and teammates.

1️⃣ CLAUDE.md = Repo Memory (keep it short)

This is the north star file.

Not a knowledge dump. Just:

• Purpose (WHY)
• Repo map (WHAT)
• Rules + commands (HOW)

If it gets too long, the model starts missing important context.

Claude needs 4 things at all times:

• the why → what the system does
• the map → where things live
• the rules → what’s allowed / not allowed
• the workflows → how work gets done

The Anatomy of a Claude Code Project 👇

Most people treat CLAUDE.md like a prompt file.

That’s the mistake.

If you want Claude Code to feel like a senior engineer living inside your repo, your project needs structure.

#Agentic #AI #Claude

If you enjoy system design, infrastructure, and data flow — engineering may suit you.
If you enjoy analysis, modeling, and problem-solving with algorithms — science may be your path.

A Data Scientist analyzes data, builds models, applies statistics, and translates patterns into actionable insights. They focus on prediction, experimentation, and business impact.

A Data Engineer designs pipelines, manages large-scale systems, ensures data reliability, and works heavily with cloud and distributed frameworks. They focus on performance, scalability, and architecture.

Data Engineer vs. Data Scientist: What’s the Difference?

One builds the data foundation.
The other turns data into intelligence.

- Using coding agents to increase the speed at which they build pipelines
- Crushing data siloes with data lakehouse architectures like Iceberg and Delta. Getting the entire company to agree upon business definitions

Data engineering is one of the few "safe" roles in the coming decade!

Data engineers in 2030 are:
- Able to handle all types of data: structured, semi-structured, and unstructured
- Integrating private data into AI in a privacy-compliant and efficient way using multi-tenant architectures

Things like Claude Code will make "building pipelines" easier, but data engineering is so much more than building pipelines!

Data engineering is projected to grow faster than AI engineering over the next decade, according to the World Economic Forum!

AI is not going to replace data engineering; it will make it increasingly more valuable!

- Typically 30–60% fewer tokens than JSON1
- Explicit lengths and fields enable validation
- Removes redundant punctuation (braces, brackets, most quotes)
- Indentation-based structure, like YAML, uses whitespace instead of braces
- Tabular arrays: declare keys once, stream data as rows

JSON:

{
"users": [
{ "id": 1, "name": "Alice", "role": "admin" },
{ "id": 2, "name": "Bob", "role": "user" }
]
}

TOON:

users[2]{id,name,role}:
1,Alice,admin
2,Bob,user

Token-Oriented Object Notation (TOON) is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input as a lossless, drop-in representation of JSON data.

#dataengineering #llm