If you have worked in the data space, you would have heard the term Metadata. It is used as a catch-all term. Here are a few things to think about when someone mentions Metadata 👇
#data #dataengineering #metadata #dataops
#data #dataengineering #metadata #dataops
1. Orchestration: Time of run, re-run information, pipeline structure, the execution time for the pipeline, pipeline failure times, etc
2. Data processing: Input parameters, failure stack trace, number of rows processed, number of rows in output, number of discarded rows, etc
2. Data processing: Input parameters, failure stack trace, number of rows processed, number of rows in output, number of discarded rows, etc
3. Data quality: Mean/sum/avg, etc. for numerical columns, available values for enum columns, etc. (think dataframe.describe in pandas)
Sometimes semantic information about the data might also be referred to as Metadata. Make sure to understand what one means when they say Metadata.
That's a wrap!
If you enjoyed this thread:
1. Follow me @startdataeng for more of these
2. RT the tweet below to share this thread with your audience
If you enjoyed this thread:
1. Follow me @startdataeng for more of these
2. RT the tweet below to share this thread with your audience
https://twitter.com/startdataeng/status/1612448003368882176
Learning data engineer can be tough, but it doesn't have to be!
Sign up for our free "Data Engineering 101" & learn about DE core concepts, building scalable & resilient systems, best practices, data modeling, and building real-world projects.
startdataengineering.com/email-course/
Sign up for our free "Data Engineering 101" & learn about DE core concepts, building scalable & resilient systems, best practices, data modeling, and building real-world projects.
startdataengineering.com/email-course/
• • •
Missing some Tweet in this thread? You can try to
force a refresh
