Read on Twitter
How to use ChatGPT to speed up data pipeline dev in a few easy steps:
1. Supply ChatGPT with your input schemas. Paste your CREATE TABLE statements directly. ChatGPT knows how to parse these and make sense of the fields inside
1/4
1. Supply ChatGPT with your input schemas. Paste your CREATE TABLE statements directly. ChatGPT knows how to parse these and make sense of the fields inside
1/4
2. Specify what type of analytical pattern you want to do on this data. ChatGPT understands aggregation (for GROUP BY queries), cumulation (for FULL OUTER JOIN queries), enrichment (for JOIN queries) and slowly changing dimension patterns.
2/4
2/4
3. ChatGPT needs more context on WHICH fields it should apply these patterns to. If you don’t supply that, it’ll give you flaming garbage.
Example: “apply slowly changing dimension type 2 pattern to fields age, gender, and phone os”
3/4
Example: “apply slowly changing dimension type 2 pattern to fields age, gender, and phone os”
3/4
4. Make sure you include other important keywords about your output schemas like “make sure to partition by date” and “make this pipeline idempotent for easier backfills”
These nudges allow ChatGPT to incorporate best practices into the pipeline easily.
#dataengineering
4/4
These nudges allow ChatGPT to incorporate best practices into the pipeline easily.
#dataengineering
4/4
• • •
Missing some Tweet in this thread? You can try to
force a refresh
