Discover and read the best of Twitter Threads about #BigQuery
Most recents (16)
I have noticed an increasing amount of poor practices floating around regarding connecting #BigQuery with Looker/Data Studio that could cost you.
My top suggestions:
- utilize _table_suffix
- build reporting tables
- BI Engine
- Materialized Views
- Caching
Details in 🧵
My top suggestions:
- utilize _table_suffix
- build reporting tables
- BI Engine
- Materialized Views
- Caching
Details in 🧵
1/5 When working with sharded tables take advantage of the _table_suffix. It will potentially save you from querying a column and querying multiple tables.
Good: SELECT _table_suffix as date, userId, location... From table_*
Bad: SELECT date, userId, location... FROM table_*
Good: SELECT _table_suffix as date, userId, location... From table_*
Bad: SELECT date, userId, location... FROM table_*
2/5 Reporting Tables provide a great step between raw data and viz tools. Especially when data does not get updated as often. In the BQ UI you can easily schedule their creation or use a combination of cloud logging, pub/sub and functions for event driven creation.
1/ Looking for a truly trustless tech stack for sourcing valid data? Look no further than the KYVE Network Stack, providing:
♾️ Data immutability
✅ Decentralized validation
💻 Customizable implementation
💫 & much more
Let’s break this down 🧵👇
♾️ Data immutability
✅ Decentralized validation
💻 Customizable implementation
💫 & much more
Let’s break this down 🧵👇
2/ With KYVE’s mission to solve today’s #Web3 data validity & access roadblocks, our team started building special tooling around its data lake...
🛠️ Ultimately creating a tech stack that enables fast & secure Web3 building via a single trustless data platform.
🛠️ Ultimately creating a tech stack that enables fast & secure Web3 building via a single trustless data platform.
3/ 🌐 Starting with the core of KYVE Network, KYVE…
KYVE is our #Web3 data lake, which can fetch any type of raw, deterministic data, then validate it & store it onto any preferred storage platform in a decentralized way.
KYVE is our #Web3 data lake, which can fetch any type of raw, deterministic data, then validate it & store it onto any preferred storage platform in a decentralized way.
Now that we got your attention... check this out: Google Cloud is running a block-producing @solana validator to participate in and validate the network.
Did you think we only had one announcement? Think again.
Google Cloud is working with @solana to bring Blockchain Node Engine to the Solana chain next year, so it will be easy for anyone to launch a dedicated Solana node in the cloud ↓ cloud.google.com/blog/products/…
Google Cloud is working with @solana to bring Blockchain Node Engine to the Solana chain next year, so it will be easy for anyone to launch a dedicated Solana node in the cloud ↓ cloud.google.com/blog/products/…
Can you imagine serverless Spark + BigQuery together? 🤯
Forget about managing clusters and tuning infrastructure if your job is to focus on create business value.
👇
🧵1/6
#googlecloud #bigquery #spark #dataengineering
Forget about managing clusters and tuning infrastructure if your job is to focus on create business value.
👇
🧵1/6
#googlecloud #bigquery #spark #dataengineering
Why Serverless Spark?
💡 Developers can focus on code and logic. They do not need to manage clusters or tune infrastructure. They submit #Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job.
🧵2/6
#googlecloud #bigquery #gcp
💡 Developers can focus on code and logic. They do not need to manage clusters or tune infrastructure. They submit #Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job.
🧵2/6
#googlecloud #bigquery #gcp
💡 Data engineering teams do not need to manage and monitor infrastructure for their end users. They are freed up to work on higher value #dataengineering functions.
💡 Pay only for the job duration, vs paying for infrastructure time.
🧵3/6
#googlecloud #bigquery #spark
💡 Pay only for the job duration, vs paying for infrastructure time.
🧵3/6
#googlecloud #bigquery #spark
A Thread of Google Cloud resources 💡
🧵
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
🧵
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
Google Cloud blog
cloud.google.com/blog/products
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
cloud.google.com/blog/products
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
Google Cloud big data product list
cloud.google.com/products/big-d…
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
cloud.google.com/products/big-d…
#gcp #google #googlecloud #googlecloudplatform #dataengineering #datascience #dataanalytics #cloud #CloudComputing
1/ A few things will happen as #web3 matures, attracting more developers to build on DLT-based storage.
The general pattern is that digital assets will begin to blend with existing content, behaviors, and even the physical world.
Here's some evidence...
The general pattern is that digital assets will begin to blend with existing content, behaviors, and even the physical world.
Here's some evidence...
2/ Supporting the thesis is @Twitter's support for verifying NFT PFPs held in $ETH wallets.
This shows how #web2 centralized and #web3 decentralized content will be blended together.
This shows how #web2 centralized and #web3 decentralized content will be blended together.
3/ For holiday gifts this year, @YouTube gave @flow_blockchain #NFT + physical display pieces to top content creators:
art.nftbridge.com/tokens/90516
flowscan.org/transaction/60…
CC @DashieXP @RichJWidmann
art.nftbridge.com/tokens/90516
flowscan.org/transaction/60…
CC @DashieXP @RichJWidmann
Après des mois de travail à distance imposé, notre bureau #Montpellier se réinvente pour accompagner notre transition sur le modèle d'Hybrid Workplace : nouveaux espaces de collaboration et de visio-conf, nouveaux espaces de travail #flexoffice et nouvelle déco ! 🔽 thread !
Le télétravail existe depuis toujours à Hawk, mais force est de constater que durant ces deux dernières années il a pris une part prépondérante dans notre quotidien. Notre organisation future sera un modèle hybride où télétravail, présentiel et remote cohabiteront harmonieusement
Le flex office, pour ou contre ? Chez Hawk la réponse a été quasi-unanime : POUR 👍 ! Parfait pour booster le foisonnement d'idées et favoriser l'émergence de solutions innovantes.
Et pour soutenir ce changement, quoi de mieux que des nouveaux espaces de travail modernisés ?
Et pour soutenir ce changement, quoi de mieux que des nouveaux espaces de travail modernisés ?
A tool to build any #GoogleDataStudio formula. Where possible it will suggest formulas for #GoogleSheets, #BigQuery, and #AppsScript.
Is it too ambitious? 🤔
🧪Work in progress. 👇
Is it too ambitious? 🤔
🧪Work in progress. 👇
✅ All aggregation functions are working
⏭️ Next - arithmetic functions.
⏭️ Next - arithmetic functions.
🧪 progress with arithmetic functions......
Modern #data platforms are emerging as the answer to the holy grail: creating a truly data-driven organization. With this, “how” we use data has changed.
Many vendors today brand themselves as the be-all and end-all... but this isn’t true. 🙅
Many vendors today brand themselves as the be-all and end-all... but this isn’t true. 🙅
It’s impossible to work today with a single modern data platform from one vendor! A modern #data platform is a collection of tools and processes. 🧰
In this thread, I’ll break down what a modern data platform means in practice today. 2/n
In this thread, I’ll break down what a modern data platform means in practice today. 2/n
Today, data platforms have basic building blocks that look something like this. 👇 3/n
towardsdatascience.com/the-building-b…
towardsdatascience.com/the-building-b…
I'm an avid reader of @saxena_puru investment tweets and am seeing tons of interest lately in @SnowflakeDB since $SNOW IPO is coming this week. I want to help people understand the product from my perspective of actually using it vs it's competitors. <THREAD>
On the PRs and web articles people cite $SNOW's architecture of decoupled compute vs storage as a moat against competition. This is simply false as all 3 cloud providers @Azure @googlecloud and @awscloud today offer data warehousing engines that decouple compute and storage.
Another factor mentioned as a moat is $SNOW's capability to easily handle "unstructured" data, usually meaning loosely structured text (JSON, logs, etc). Again, all 3 cloud providers offer robust capabilities in this regard.
今天看到 @haoel 说Google Docs的Post，联想昨天晚上喷团队里的同学把Jira用成了To-Do List，颇有点不吐不快。那就花点时间说说根据过去几年各种公司使用的SAAS/工具的体验。有些工具，真的是用过了就回不去的。
#SAAS #Productivity
#SAAS #Productivity
基本上，以下这些SAAS/工具是用了之后就回不去的
1. G-Suite全家桶
2. Atlassian全家桶（Confluence+Jira+…）
3. Google Cloud里围绕BigQuery的整套系统
4. Cloudflare
5. NewRelic
#SAAS #Productivity
1. G-Suite全家桶
2. Atlassian全家桶（Confluence+Jira+…）
3. Google Cloud里围绕BigQuery的整套系统
4. Cloudflare
5. NewRelic
#SAAS #Productivity
除了基本的办公协同之外，如果你真的重度使用“数字化系统”来管理公司的业务，你会发现G-Suite真的是太强大了。
第一个就是统一的权限管理，大部分SAAS系统都有Google登陆的支持。公司内部的系统，也只支持Google登陆。这样所有同学入职离职，都只需要开一个G-Suite账号就好了，不用入职开账号就一天。
第一个就是统一的权限管理，大部分SAAS系统都有Google登陆的支持。公司内部的系统，也只支持Google登陆。这样所有同学入职离职，都只需要开一个G-Suite账号就好了，不用入职开账号就一天。
بمناسبة اليوم الوطني راح أهديكم سلسلة تغريدات ألخص فيها تجربتي مع منصة #فايربيس @Firebase
فايربيس هي منصة للمبرمجين وخصوصا مطوري التطبيقات تملكها قوقل @Google وهذا رابط المنصة firebase.google.com
فايربيس هي منصة للمبرمجين وخصوصا مطوري التطبيقات تملكها قوقل @Google وهذا رابط المنصة firebase.google.com
لماذا فايربيس؟ فايربيس عبارة عن منصة باكاند backend متكاملة توفر لمطوري التطبيقات عدة خدمات على رأسها ١) ادارة المستخدمين firebase authentication و ٢) قاعدة بيانات #فايرستور #Firestore و ٣) تحليلات فايربيس اناليتكس firebase analytics لتحليل سلوك مستخدمي التطبيق
ايش يفرق فايربيس عن خدمات أخرى مثل قوقل كلاود او أمازون كلاود او غيرها؟
الفرق الرئيسي والأساسي هو انك تحصل على الباكاند من فايربيس كخدمة backend as a service او #BaaS بدون ما تكتب ولا سطر كود خارج تطبيقك. هذا لا يعني انك راح تستغني بشكل كامل عن كتابة كود باكاند زي ما راح نشوف
الفرق الرئيسي والأساسي هو انك تحصل على الباكاند من فايربيس كخدمة backend as a service او #BaaS بدون ما تكتب ولا سطر كود خارج تطبيقك. هذا لا يعني انك راح تستغني بشكل كامل عن كتابة كود باكاند زي ما راح نشوف
Hey all, I’m doing a weekly twitter standup, mostly for myself. But if you or anyone else would like to join, feel free to ping me. I’m thinking a weekly basis won’t be too time consuming. Pick a day of the week, and I’ll @ you to remind you. #weeklytwitterstandup
Did last week:
✅ finally built script to track usage of my servers. It’s a python script uploaded to AWS lambda. The script pings my half-dozen servers for data usage and writes the data onto 1) a MySQL server hosted on AWS RDS, 2) BigQuery database, and 3) google sheets
✅ finally built script to track usage of my servers. It’s a python script uploaded to AWS lambda. The script pings my half-dozen servers for data usage and writes the data onto 1) a MySQL server hosted on AWS RDS, 2) BigQuery database, and 3) google sheets
✅ built a dashboard on server usage data using Tableau public. I now have the dashboard scrolling on my Life Flips app.
As I'm sitting in this talk, @jrdntgn is dropping a lot of things that #BigQuery can do in a short amount of time that I am not sure one can get the magnitude of how awesome they are until some reflection (kind of a slo-mo replay).
I'll try to point some of them out in here.
I'll try to point some of them out in here.
1/ Wanted to wait until I can refer to the actual recording on YouTube (here it is ).
So, here we go ...
So, here we go ...
I'm teaching a "data science for economists" course this semester.
If you're interested in learning more about #rstats, Git(Hub), programming, databases, cloud computation, ML, etc., I'll be making all of my course material publicly available here: github.com/uo-ec607
If you're interested in learning more about #rstats, Git(Hub), programming, databases, cloud computation, ML, etc., I'll be making all of my course material publicly available here: github.com/uo-ec607
As I say in the syllabus, this course basically covers all of the things I wish I'd been taught in grad school. At the same time, I've benefited immensely from so many people making their teaching materials (and software!) publicly available. This is me trying to pay it forward.
Here's a short-cut to the lecture slides and notes: github.com/uo-ec607/lectu…
👋 Hi, friends! A couple of folks DM-ed asking about the difference between @GoogleColab and @GoogleCloud #DataLab.
📒 Both services are built with @ProjectJupyter, and look kinda similar; and both are useful for data exploration! - but that is the extent of the overlap. (1/n)
📒 Both services are built with @ProjectJupyter, and look kinda similar; and both are useful for data exploration! - but that is the extent of the overlap. (1/n)
✨📒 @GoogleColab doesn’t require any setup or other @Google products to be used; though notebooks are stored on @GoogleDrive.
It’s intended primarily for interactive use- which means some long-running background processes may be stopped. It currently only supports Python. (2/n)
It’s intended primarily for interactive use- which means some long-running background processes may be stopped. It currently only supports Python. (2/n)
✨👩🔬#DataLab is also free to use; but allows you to analyze data using @GoogleCloud resources. You can take full advantage of scalable services such as #BigQuery and Cloud ML Engine to analyze, manipulate and visualize data.
📊 Python, SQL, and JavaScript are supported. (3/n)
📊 Python, SQL, and JavaScript are supported. (3/n)
