Discover and read the best of Twitter Threads about #datawarehouse
Most recents (2)
How is #DuckDb going to be used inside a company? My head can't think beyond it being used for local development. Can it really replace a DWH without being distributed?
If it does go beyond local, I do have a few things to ponder upon, 🧵
If it does go beyond local, I do have a few things to ponder upon, 🧵
1. Search and Discovery: Where are users going to find artifacts inside the DuckDB cluster (eventually)? We'll need a catalog interface to surface this portion. The metadata of the artifacts in the data model (database, table) needs to be consistent across the cluster.
2. How is privacy handled? What if someone downloads sensitive data onto their laptop and it is stolen thus opening up a data breach? My paranoia comes from working on Data infra for GDPR, and CCPA. Access control should be implemented and enforced.
What is the difference between a Data Engineer and a Data Architect?
🧵[1/x]
🧵[1/x]
A data engineer looks at the immediate set of requirements and works towards that. In other words, data engineers build, rebuild, and tear down. ⚒
Need a new field in the report? Let's just build the whole thing. ⚒
🧵[2/x]
Need a new field in the report? Let's just build the whole thing. ⚒
🧵[2/x]
Data Architects think ahead in terms of capacity planning. X years from now, Y will happen, so we'll need to consider Z. In other words, Data Architects look at the full requirements and build it once.😎
This means less waste of money for the company in the long run.
🧵[3/x]
This means less waste of money for the company in the long run.
🧵[3/x]
