Thread by @ahmedmayad

Mr. #BigQuery himself - @jrdntgn - on stage.

As I'm sitting in this talk, @jrdntgn is dropping a lot of things that #BigQuery can do in a short amount of time that I am not sure one can get the magnitude of how awesome they are until some reflection (kind of a slo-mo replay).

I'll try to point some of them out in here.

1/ Wanted to wait until I can refer to the actual recording on YouTube (here it is ).

So, here we go ...

2/ Let's start with the obvious: the traditional petabyte query which keeps getting faster (this year it stands at 11 seconds, with @jrdntgn pinning it on us to get to under a second. Thanks for the pressure dude! 🙄).

3/ The demo starts here: .

The thread here:

https://twitter.com/ahmedmayad/status/1116744490255917061

expands on why this is awesome!

4/ Next, take #BigQuery #ML (#BQML in case you see that abbreviation). Starts here:

It is mentioned casually that the test case we used internally for the new Matrix Factorization task was the Netflix recommendation challenge!

5/ True, ML has advanced a bunch since then. But, the fact that a platform like #BigQuery can give users like me with little ML training the capability to achieve in minutes, out of the box, using just SQL, what teams competed for a $1M prize just 10 years ago is awesome!

6/ Finally (for this thread), the #HighThroughput streaming demo. It starts here: .

Streaming isn't new to #BigQuery, nor is it unique to us. So, what's the big deal here?

7/ Listen carefully to the setup of the demo: "In the past, people have kinda run into some streaming limits in #BigQuery, and we are constantly working on breaking through those limits!"

For reference, these are 100k rows/s, 100 MB/s (by default).

8/ So, this isn't really about what we are capable of (you can assume that in aggregate we are indeed capable of a lot more), this is a demo of what #you will be capable of at some point!

9/ The demo app had 10 clients streaming into a table in #BigQuery. Each one pushing ~2GB/s (see: ). Doing the math, this is an application pushing ~#20GB/s total. As for rows/s, in aggregate, this application was streaming ~#2M rows/s.

10/ This is basically about #100X the current default throughput, and about #10X the default rows/s.

Then, @jrdntgn goes on to demonstrate how #BigQuery - in real time - is capable of picking up a single event as the application is pushing a torrent of them simultaneously.

11/ At some point @jrdntgn mentions modestly that "ok, perhaps lots of systems can handle that sort of scale". Maybe #BigQuery can, but I don't think any other allows you to do this without a ton of tuning and babysitting. We aspire to make it happen out of the box, for all.

12/12 Finally, there is an #EasterEgg in that demo. If you look closely, you should be able to get out internal name for this new awesomeness! 😃

Please watch the preso, and all the other ones on the @GCPcloud channel. Thx for being the greatest user base for #BigQuery.