David Regalado Profile picture
Mar 27 โ€ข 7 tweets โ€ข 7 min read
Can you imagine serverless Spark + BigQuery together? ๐Ÿคฏ

Forget about managing clusters and tuning infrastructure if your job is to focus on create business value.

๐Ÿ‘‡

๐Ÿงต1/6

#googlecloud #bigquery #spark #dataengineering
Why Serverless Spark?

๐Ÿ’ก Developers can focus on code and logic. They do not need to manage clusters or tune infrastructure. They submit #Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job.

๐Ÿงต2/6

#googlecloud #bigquery #gcp
๐Ÿ’ก Data engineering teams do not need to manage and monitor infrastructure for their end users. They are freed up to work on higher value #dataengineering functions.

๐Ÿ’ก Pay only for the job duration, vs paying for infrastructure time.

๐Ÿงต3/6

#googlecloud #bigquery #spark
Serverless Spark through BigQuery

BigQuery is adding a unified interface for data analysts to write SQL or PySpark. That Preview is now live, you can request access through the signup form.

๐Ÿ‘‰ lnkd.in/eeB2gqH6

๐Ÿงต4/6

#googlecloud #bigquery #spark #dataengineering
What is BigQuery again?

Is a serverless, highly scalable, and cost-effective ๐—บ๐˜‚๐—น๐˜๐—ถ๐—ฐ๐—น๐—ผ๐˜‚๐—ฑ data warehouse designed for business agility.

๐Ÿ‘‰Learn BigQuery in a minute: lnkd.in/eMS25HAJ

๐Ÿงต5/6

#googlecloud #bigquery #spark #dataengineering
What's next?

Watch for the availability of serverless Spark through Vertex AI workbench for data scientists, and Dataplex for data analysts, in the coming months.

๐Ÿ‘‰I tweet about all things data-related. Follow me for more.

๐Ÿงต6/6

#googlecloud #bigquery #spark #dataengineering

โ€ข โ€ข โ€ข

Missing some Tweet in this thread? You can try to force a refresh
ใ€€

Keep Current with David Regalado

David Regalado Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @thecodemancer_

Mar 18
Bored employees are super disengaged, prone to conflict and suffer burnout at a higher rate.

Harvard Business Review found employees who are learning at work experience less anxiety and stress. They are also more ethical than bored workers on autopilot.

๐Ÿงต1/7

#manager #career
All employees deserve to learn, bored, engaged or somewhere in between. Itโ€™s your job as a manager to make that happen. Here are three ideas to create a learning strategy and become a manager whose people wonโ€™t leave:

๐Ÿงต2/7

#manager #career
1. Plan to learn something new every week.

Invite your employees to schedule 10 minutes on their calendars every day to learn something new. Then, weekly, ask your employees about what theyโ€™ve learned.

๐Ÿงต3/7

#manager #career #learning
Read 7 tweets
Mar 16
Is Infrastructure as SQL (IaSQL) a thing? ๐Ÿค”

๐Ÿงต1

#dataengineering #sql #IaSQL #cloud
๐—›๐—ผ๐˜„ ๐—œ๐—ฎ๐—ฆ๐—ค๐—Ÿ ๐˜„๐—ผ๐—ฟ๐—ธ๐˜€

IaSQL is an open-source SaaS that models cloud infrastructure as data by maintaining a 2-way connection between a AWS account and a hosted PostgreSQL database. โ˜

๐Ÿงต2
๐—ช๐—ต๐˜† ๐—œ๐—ฎ๐—ฆ๐—ค๐—Ÿ?

๐Ÿ‘‰ ๐˜๐˜ฎ๐˜ฑ๐˜ฐ๐˜ณ๐˜ต ๐˜ฆ๐˜น๐˜ช๐˜ด๐˜ต๐˜ช๐˜ฏ๐˜จ ๐˜ค๐˜ญ๐˜ฐ๐˜ถ๐˜ฅ ๐˜ช๐˜ฏ๐˜ง๐˜ณ๐˜ข๐˜ด๐˜ต๐˜ณ๐˜ถ๐˜ค๐˜ต๐˜ถ๐˜ณ๐˜ฆ

Connect an AWS account to a hosted IaSQL DB to automatically backfill the database with your existing cloud resources. No need to redefine or reconcile existing infrastructure.

๐Ÿงต3
Read 6 tweets
Feb 26
The evolution of data processing frameworks.

Knowing how these frameworks have evolved can help you understand the typical problems that arise, and how they're addressed.

As the Internet grew, Google invented new data processing methods.

๐Ÿงต

#GCP #google @google @googlecloud
In 2002, Google created GFS, or the Google File System to handle sharding and storing petabytes of data at scale.

GFS is a foundation for cloud storage, and also for what would become BitQuery managed storage.

๐Ÿงต

#GCP #google @google @googlecloud
One of the next challenges was to figure out how to index the exploding volume of content on the Web.

To solve this, in 2004 @Google invented a new style of data processing (MapReduce) to manage large scale data processing across large clusters of commodity servers.

๐Ÿงต

#GCP
Read 9 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

:(