Thread by @prukalpa

Modern #data platforms are emerging as the answer to the holy grail: creating a truly data-driven organization. With this, “how” we use data has changed.

Many vendors today brand themselves as the be-all and end-all... but this isn’t true. 🙅

It’s impossible to work today with a single modern data platform from one vendor! A modern #data platform is a collection of tools and processes. 🧰

In this thread, I’ll break down what a modern data platform means in practice today. 2/n

Today, data platforms have basic building blocks that look something like this. 👇 3/n

towardsdatascience.com/the-building-b…

The center stone is a modern #data storage and processing layer. This can be 👇

Data Warehouse 👉 @SnowflakeDB, #Redshift, #BigQuery

Data Lake 👉 Object storage (#AWS S3/ #GCS) + Query Layer (@trinodb, @starburstdata, @dremio) + Metadatastore (Glue, @HiveData) or @databricks

The #Data Ingestion Layer: Getting data into the data warehouse/ lake from dozens of sources and ingest it into the core data storage layer.

#SaaS tools 👉 @fivetran, @HevoData, and @stitch_data and also the good old custom #Python jobs and Airflow DAGs.

Data Transformation: Taking raw data and converting it into the final “golden layer”.

With data warehouses 👉 @getdbt (SQL first), @matillion (no code)

With an orchestration engine 👉 @ApacheAirflow + #Python, R, or #SQL

While BI tools have been in the picture for a while, the tools today are more “self-served” to fit within a larger modern data platform.

Visualization/ BI 👉 @tableau, @LookerData, @Sisense, @metabase, @apachesuperset

#BusinessIntelligence #dataviz

As #metadata itself becomes big data, there are Metadata Management/ Data Catalogs built on the premise of embedded collaboration (think @github, @figmadesign, @SlackHQ, @NotionHQ, @Superhuman) 🤝

#SaaS 👉 @AtlanHQ
#OpenSource 👉 @amundsenio, #Datahub

#BigData #datacatalog

Data Access Governance: Tools that can act as an entitlement engine to apply #privacy and security policies across the #data stack. 🔐

#SaaS 👉 @ImmutaData, @okerainc, @privacera,

#OpenSource 👉 @ApacheRanger

#datagovernance #dataprotection