Thread by @tyler_agg

there's so much content on how to build AI agents, but no one ever talks about the data engineering pipelines that support them

here's a thread going over the basics of data engineering:

the goal of data engineering:

- extract data from various sources
- transform it into structured format
- load into a data warehouse like Snowflake

and this structured data often is used context for AI systems to make personalized recommendations

why this matters for AI:

to build personalized AI systems, you need clean, structured data

as the more structured and labeled it is, the more granular/accurate context we can retrieve for an AI system

step 1: extract data from sources

where is your data coming from?

- Google Sheets (pull via API)
- websites (web scraping)
- third-party APIs
- existing databases

this part depends entirely on what data you're trying to collect

step 2: transform the raw data

raw data is messy and unstructured - contract text, website content, whatever

you need to:

- clean up missing values
- add structure to make it consistent
- standardize formats across different sources

AI is often used in transformation pipelines as well

an example I’ve done: extracting entities from contract text like specific clauses, start/end dates, party names

raw text isn't in clean tabular format obviously

so we use to AI to pull information from unstructured documents to put them in a tabular format

(like think of a google sheet with a column for each clause and key entity we want to collect for example)

step 3: load the data into a data warehouse

push the cleaned, structured data into Snowflake or similar warehouse

and now you have tables of organized data that you can easily query and use as context

the tools involved:

- Python for scripting
- AWS for cloud infrastructure
- Airflow for workflow orchestration
- Snowflake for data warehousing
- DBT for data transformations

there’s an entire tech stack that comes with this shit and that’s why data engineers get paid good af

note as well why reliability is critical af with data pipelines is

if if breaks anywhere here, every AI system built on top of it is cooked

bad data here = broken systems downstream

to recap data engineering basics:

- extract data from various sources (APIs, scraping, databases)
- transform raw data into structured format (clean, standardize, extract entities)
- load into data warehouse
- build reliably with error handling and monitoring