Share this page!

Enter Twitter Thread URL to Unroll

Needs to be the full URL like: https://twitter.com/threadreaderapp/status/1644127596119195649

How to get URL link on Twitter App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Start Data Engineering Profile picture
@startdataeng
May 22 • 4 tweets • 4 min read Twitter logo Read on Twitter
Data engineers work with multiple systems & it's crucial to understand DevOps. Shown below are a few DevOps concepts to familiarize oneself with:

1. Docker: docs.docker.com/get-started/
2. Kubernetes: kubernetes.io/docs/concepts/…
3. CI/CD: resources.github.com/ci-cd/

#dataengineering
#data
4. IAC: pulumi.com/what-is/what-i…
5. Monitoring: atlassian.com/devops/devops-…
6. Access control: techtarget.com/searchsecurity…
7. Key management: aws.amazon.com/kms/?c=sc&sec=…
8. Encrypted connections: dev.mysql.com/doc/refman/8.0…
That's a wrap!

If you enjoyed this thread:

1. Follow me @startdataeng for more of these
2. RT the tweet below to share this thread with your audience
Learning data engineer can be tough, but it doesn't have to be!

Sign up for our free "Data Engineering 101" & learn about DE core concepts, building scalable & resilient systems, best practices, data modeling, and building real-world projects.

startdataengineering.com/email-course/

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Start Data Engineering

Start Data Engineering Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @startdataeng

Start Data Engineering Profile picture
May 23
Testing your pipelines before merging is crucial to ensure they do not fail in production. However, testing data pipelines is complex (and expensive) due to the data size, confidentiality, and time it takes to test a data pipeline.
🧵
#data #dataengineering #testing #dataops
Here are a few ways to get data for your tests:

1. Copying data: An exact copy of the prod data for testing will ensure that our changes are not breaking the pipeline. This is expensive! You can use a part of data for testing, accepting possible edge case misses.
2. Data git: Projects like Nessie and LakeFS can help set up different environments without replicating entire data.
Read 7 tweets
Start Data Engineering Profile picture
Jan 9
If you have worked in the data space, you would have heard the term Metadata. It is used as a catch-all term. Here are a few things to think about when someone mentions Metadata 👇

#data #dataengineering #metadata #dataops
1. Orchestration: Time of run, re-run information, pipeline structure, the execution time for the pipeline, pipeline failure times, etc

2. Data processing: Input parameters, failure stack trace, number of rows processed, number of rows in output, number of discarded rows, etc
3. Data quality: Mean/sum/avg, etc. for numerical columns, available values for enum columns, etc. (think dataframe.describe in pandas)
Read 6 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

Send Email!

:(