, 13 tweets, 3 min read
My Authors
Read all threads
Not getting results you want from data lake? You may think of trying data lake in the cloud - but it's the same paradigm with same issues. @zhamakd at #QConSF
The problem is that the layers in data lake design map to technical components. This doesn't work - change is orthogonal.
Data engineers are siloed and stuck between teams with different incentives and priorities.
In a nutshell
We need to slice the data problem around business domains
Pipelines still exist, but they are an implementation detail of those domain data products.
Think of domain data as a product, and create a great product experience. But for that, they need to be discoverable, accessible and trustworthy. APIs have SLOs, and data needs the same.
The KPI for the product is the time to discover and consume the data for the domain. Can we decrease it?
Shared concerns become shared, self-serve infra. Things like setting access controls - right now they are different for every system.
We need common way for data products to talk to each other. Like APIs talk in HTTP. Enable interoperability and discoverability.
New ownership model
What it looks like in reality
Data products hide the pipelines and provide ports.
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Gwen (Chen) Shapira

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!