Discover and read the best of Twitter Threads about #bigquery

Most recents (16)

I have noticed an increasing amount of poor practices floating around regarding connecting #BigQuery with Looker/Data Studio that could cost you.

My top suggestions:
- utilize _table_suffix
- build reporting tables
- BI Engine
- Materialized Views
- Caching

Details in 🧵
1/5 When working with sharded tables take advantage of the _table_suffix. It will potentially save you from querying a column and querying multiple tables.

Good: SELECT _table_suffix as date, userId, location... From table_*
Bad: SELECT date, userId, location... FROM table_* 2 queries doing the exact s...
2/5 Reporting Tables provide a great step between raw data and viz tools. Especially when data does not get updated as often. In the BQ UI you can easily schedule their creation or use a combination of cloud logging, pub/sub and functions for event driven creation.
Read 7 tweets
1/ Looking for a truly trustless tech stack for sourcing valid data? Look no further than the KYVE Network Stack, providing:

♾️ Data immutability
✅ Decentralized validation
💻 Customizable implementation
💫 & much more

Let’s break this down 🧵👇
2/ With KYVE’s mission to solve today’s #Web3 data validity & access roadblocks, our team started building special tooling around its data lake...

🛠️ Ultimately creating a tech stack that enables fast & secure Web3 building via a single trustless data platform.
3/ 🌐 Starting with the core of KYVE Network, KYVE…

KYVE is our #Web3 data lake, which can fetch any type of raw, deterministic data, then validate it & store it onto any preferred storage platform in a decentralized way.
Read 9 tweets
Hey @aeyakovenko 👋

Should we tell our followers the big news?
Now that we got your attention... check this out: Google Cloud is running a block-producing @solana validator to participate in and validate the network.
Did you think we only had one announcement? Think again.

Google Cloud is working with @solana to bring Blockchain Node Engine to the Solana chain next year, so it will be easy for anyone to launch a dedicated Solana node in the cloud ↓…
Read 5 tweets
Can you imagine serverless Spark + BigQuery together? 🤯

Forget about managing clusters and tuning infrastructure if your job is to focus on create business value.



#googlecloud #bigquery #spark #dataengineering
Why Serverless Spark?

💡 Developers can focus on code and logic. They do not need to manage clusters or tune infrastructure. They submit #Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job.


#googlecloud #bigquery #gcp
💡 Data engineering teams do not need to manage and monitor infrastructure for their end users. They are freed up to work on higher value #dataengineering functions.

💡 Pay only for the job duration, vs paying for infrastructure time.


#googlecloud #bigquery #spark
Read 7 tweets
1/ A few things will happen as #web3 matures, attracting more developers to build on DLT-based storage.

The general pattern is that digital assets will begin to blend with existing content, behaviors, and even the physical world.

Here's some evidence...
2/ Supporting the thesis is @Twitter's support for verifying NFT PFPs held in $ETH wallets.

This shows how #web2 centralized and #web3 decentralized content will be blended together.
Read 15 tweets
Après des mois de travail à distance imposé, notre bureau #Montpellier se réinvente pour accompagner notre transition sur le modèle d'Hybrid Workplace : nouveaux espaces de collaboration et de visio-conf, nouveaux espaces de travail #flexoffice et nouvelle déco ! 🔽 thread !
Le télétravail existe depuis toujours à Hawk, mais force est de constater que durant ces deux dernières années il a pris une part prépondérante dans notre quotidien. Notre organisation future sera un modèle hybride où télétravail, présentiel et remote cohabiteront harmonieusement
Le flex office, pour ou contre ? Chez Hawk la réponse a été quasi-unanime : POUR 👍 ! Parfait pour booster le foisonnement d'idées et favoriser l'émergence de solutions innovantes.
Et pour soutenir ce changement, quoi de mieux que des nouveaux espaces de travail modernisés ?
Read 7 tweets
A tool to build any #GoogleDataStudio formula. Where possible it will suggest formulas for #GoogleSheets, #BigQuery, and #AppsScript.

Is it too ambitious? 🤔

🧪Work in progress. 👇
✅ All aggregation functions are working

⏭️ Next - arithmetic functions.
🧪 progress with arithmetic functions......
Read 3 tweets
Modern #data platforms are emerging as the answer to the holy grail: creating a truly data-driven organization. With this, “how” we use data has changed.

Many vendors today brand themselves as the be-all and end-all... but this isn’t true. 🙅
It’s impossible to work today with a single modern data platform from one vendor! A modern #data platform is a collection of tools and processes. 🧰

In this thread, I’ll break down what a modern data platform means in practice today. 2/n
Today, data platforms have basic building blocks that look something like this. 👇 3/n…
Read 9 tweets
I'm an avid reader of @saxena_puru investment tweets and am seeing tons of interest lately in @SnowflakeDB since $SNOW IPO is coming this week. I want to help people understand the product from my perspective of actually using it vs it's competitors. <THREAD>
On the PRs and web articles people cite $SNOW's architecture of decoupled compute vs storage as a moat against competition. This is simply false as all 3 cloud providers @Azure @googlecloud and @awscloud today offer data warehousing engines that decouple compute and storage.
Another factor mentioned as a moat is $SNOW's capability to easily handle "unstructured" data, usually meaning loosely structured text (JSON, logs, etc). Again, all 3 cloud providers offer robust capabilities in this regard.
Read 11 tweets
今天看到 @haoel 说Google Docs的Post,联想昨天晚上喷团队里的同学把Jira用成了To-Do List,颇有点不吐不快。那就花点时间说说根据过去几年各种公司使用的SAAS/工具的体验。有些工具,真的是用过了就回不去的。

#SAAS #Productivity

1. G-Suite全家桶
2. Atlassian全家桶(Confluence+Jira+…)
3. Google Cloud里围绕BigQuery的整套系统
4. Cloudflare
5. NewRelic

#SAAS #Productivity

Read 37 tweets
بمناسبة اليوم الوطني راح أهديكم سلسلة تغريدات ألخص فيها تجربتي مع منصة #فايربيس @Firebase
فايربيس هي منصة للمبرمجين وخصوصا مطوري التطبيقات تملكها قوقل @Google وهذا رابط المنصة
لماذا فايربيس؟ فايربيس عبارة عن منصة باكاند backend متكاملة توفر لمطوري التطبيقات عدة خدمات على رأسها ١) ادارة المستخدمين firebase authentication و ٢) قاعدة بيانات #فايرستور #Firestore و ٣) تحليلات فايربيس اناليتكس firebase analytics لتحليل سلوك مستخدمي التطبيق
ايش يفرق فايربيس عن خدمات أخرى مثل قوقل كلاود او أمازون كلاود او غيرها؟
الفرق الرئيسي والأساسي هو انك تحصل على الباكاند من فايربيس كخدمة backend as a service او #BaaS بدون ما تكتب ولا سطر كود خارج تطبيقك. هذا لا يعني انك راح تستغني بشكل كامل عن كتابة كود باكاند زي ما راح نشوف
Read 117 tweets
Hey all, I’m doing a weekly twitter standup, mostly for myself. But if you or anyone else would like to join, feel free to ping me. I’m thinking a weekly basis won’t be too time consuming. Pick a day of the week, and I’ll @ you to remind you. #weeklytwitterstandup
Did last week:
✅ finally built script to track usage of my servers. It’s a python script uploaded to AWS lambda. The script pings my half-dozen servers for data usage and writes the data onto 1) a MySQL server hosted on AWS RDS, 2) BigQuery database, and 3) google sheets
✅ built a dashboard on server usage data using Tableau public. I now have the dashboard scrolling on my Life Flips app.
Read 31 tweets
Mr. #BigQuery himself - @jrdntgn - on stage.
As I'm sitting in this talk, @jrdntgn is dropping a lot of things that #BigQuery can do in a short amount of time that I am not sure one can get the magnitude of how awesome they are until some reflection (kind of a slo-mo replay).

I'll try to point some of them out in here.
1/ Wanted to wait until I can refer to the actual recording on YouTube (here it is ).

So, here we go ...
Read 14 tweets
I'm teaching a "data science for economists" course this semester.

If you're interested in learning more about #rstats, Git(Hub), programming, databases, cloud computation, ML, etc., I'll be making all of my course material publicly available here:
As I say in the syllabus, this course basically covers all of the things I wish I'd been taught in grad school. At the same time, I've benefited immensely from so many people making their teaching materials (and software!) publicly available. This is me trying to pay it forward.
Here's a short-cut to the lecture slides and notes:…
Read 21 tweets
👋 Hi, friends! A couple of folks DM-ed asking about the difference between @GoogleColab and @GoogleCloud #DataLab.

📒 Both services are built with @ProjectJupyter, and look kinda similar; and both are useful for data exploration! - but that is the extent of the overlap. (1/n)
✨📒 @GoogleColab doesn’t require any setup or other @Google products to be used; though notebooks are stored on @GoogleDrive.

It’s intended primarily for interactive use- which means some long-running background processes may be stopped. It currently only supports Python. (2/n)
✨👩‍🔬#DataLab is also free to use; but allows you to analyze data using @GoogleCloud resources. You can take full advantage of scalable services such as #BigQuery and Cloud ML Engine to analyze, manipulate and visualize data.

📊 Python, SQL, and JavaScript are supported. (3/n)
Read 4 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!