Discover and read the best of Twitter Threads about #datawarehouse

Most recents (2)

How is #DuckDb going to be used inside a company? My head can't think beyond it being used for local development. Can it really replace a DWH without being distributed?

If it does go beyond local, I do have a few things to ponder upon, ๐Ÿงต
1. Search and Discovery: Where are users going to find artifacts inside the DuckDB cluster (eventually)? We'll need a catalog interface to surface this portion. The metadata of the artifacts in the data model (database, table) needs to be consistent across the cluster.
2. How is privacy handled? What if someone downloads sensitive data onto their laptop and it is stolen thus opening up a data breach? My paranoia comes from working on Data infra for GDPR, and CCPA. Access control should be implemented and enforced.
Read 7 tweets
What is the difference between a Data Engineer and a Data Architect?

๐Ÿงต[1/x]
A data engineer looks at the immediate set of requirements and works towards that. In other words, data engineers build, rebuild, and tear down. โš’

Need a new field in the report? Let's just build the whole thing. โš’

๐Ÿงต[2/x]
Data Architects think ahead in terms of capacity planning. X years from now, Y will happen, so we'll need to consider Z. In other words, Data Architects look at the full requirements and build it once.๐Ÿ˜Ž

This means less waste of money for the company in the long run.

๐Ÿงต[3/x]
Read 6 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!