Discover and read the best of Twitter Threads about #spark

Most recents (12)

Ever wondered how to manipulate big data with R?

A thread on using Spark in R👇🧵

#rstats #spark #datascience
Big data is any data which is too large to be loaded into memory all in one go. Ever tried to read a large dataset into R and everything fell over or slowed waaaaaay down? This is where Spark comes in.

1/18
Spark is an open source tool for processing big data. It splits data into partitions for processing to overcome RAM limitations and writes to disk when needed. It is complicated but Spark handles most of the difficult parts. And you can work with Spark directly from R.

2/18
Read 19 tweets
Can you imagine serverless Spark + BigQuery together? 🤯

Forget about managing clusters and tuning infrastructure if your job is to focus on create business value.

👇

🧵1/6

#googlecloud #bigquery #spark #dataengineering
Why Serverless Spark?

💡 Developers can focus on code and logic. They do not need to manage clusters or tune infrastructure. They submit #Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job.

🧵2/6

#googlecloud #bigquery #gcp
💡 Data engineering teams do not need to manage and monitor infrastructure for their end users. They are freed up to work on higher value #dataengineering functions.

💡 Pay only for the job duration, vs paying for infrastructure time.

🧵3/6

#googlecloud #bigquery #spark
Read 7 tweets
🧵1. #IPERN officially launched on #StPatricksDay 2021 so it's our 1st birthday today!🎂🚑☘ It's been quite a year - we've hosted 10 events featuring 49 speakers (31% International 🌍 65% Female 🚺). Presented at 3 conferences, secured 3 grants and launched 5 SIG! #IrishEMS
2. How it all began… we announced ourselves to the Twitter world on Feb 1st '21. In Ireland this is traditionally the start of Spring (#Imbolc) so a time for new beginnings. It’s also St. Brigid‘s Day our matron saint who Is renowned as a healer #IrishEMS tinyurl.com/2p8w2zwt
3. Our official #IPERN launch on #StPatricksDay 2021 featured a fantastic array of Irish researchers presenting "elevator pitches" of their work and also included best wishes from international friends & colleagues! tinyurl.com/2p92at99 #IrishEMS
Read 34 tweets
I realized today one of the reasons I appreciate working for a system integrator. Last time this year I was training in #Scala and #Spark. Then, I was placed in a project using #python, #pandas, #pyspark and #sql. That ended 12/31/21 and I had a short break before joining...
a pilot project working with #terraform, #aws, #boto3 with #python, #informatica, #autosys and #shellScripting. Not to mention the increasing frequency of situations that are like "Here is some technology setup that you have never seen before, here is some rudimentary docs,...
Figure it out in 1 or 2 sprints, provide deliverable artifacts and documentation on the preexisting system AND the new artifacts. Thx, k. Bai!" Oh, and teammates you joined with, they all have a shit-ton of stuff, so don't expect replies to your msgs.

Cue: 1. panic, then...
Read 6 tweets
I want to start a new thread story. Not sure how long it will be and I may use different prompts: #vss365 #vssmagic #vssnature etc etc ... I love reading about legends and folk lore, and in my fantasy writing I often create my own myths. So this will be just a fun exercise...
According to #lore &myths of old,griffins nestled on these cliffs long ago. They swooped down on the unsuspecting;carried them off never to be seen again.
Griffins were not elemental creatures,but air & earth did have dominion over them; their only enemy -fire.
#vss365 #vssmagic
"I helped your troupe avoid #shadow and #gore," said the dark wizard simply, implying that he saved us from the mythical griffins. I gaped at him. "And now for what you promised."

"No," I was careful not to get trapped in words, "I said: after you summon her."
#vss365 #vssmagic
Read 132 tweets
Discordで教えてもらったんだけど、新しいMEW使ってる場合は24のシードフレーズでメタマスク上に復元できるので、インポートとかせずに直で復元すれば無問題のようです。
つまり、いずれBifrost wallet使いたくて、シードフレーズがあるMEWのETHアドレスで #FLR のオプトイン申請した人は、メタマスクのETHアドレスで申請し直さなくてもOK。 #XRP #FLR #Spark #FlareNetworks
Bifrost wallet は接続にシードフレーズが必要で、秘密鍵では接続できないから、シードフレーズがない旧版MEWのETHアドレスとは接続できないよ〜ということです。(多分)
Read 4 tweets
Assuming you're not using aliases, a plain text search in @obsdmd for "[[page]]" within quotations pulls up the backlinks for that page. Searching for ".png" will pull up all of your pngs. There's a lot of flexibility that comes from searching strings when everything is in markup
Here I have an embedded query to find all my notes directly dealing with DSLs (domain-specific languages). I get this by searching for file names that have DSL in them. In Roam I might have had [[A [[DSL]] speeds up the author and gives them the ability to write from anywhere]] ImageImage
Here I'm trying out using namespacing in a @dendronhq style. I have an embedded search looking for pages that have Clojure and function in the title. It's not officially supported in Obsidian, but with searches embedded on pages, I'm able to do it quite flexibly ImageImage
Read 10 tweets
How can I start a career in #risk management in the financial services industry?
@GARP_Risk @PRMIA @RIMSorg @actuarynews @SOActuaries @CQFInstitute @CFAinstitute
I believe they are many degrees that can be a good value for money, provided you know where you would like to work in the long run.
In the developing world markets, employers still don't appreciate highly technical education in risk, actuarial sciences, financial engineering, quantitative economics, machine learning, etc.

For them, an #MBA is a be-all and an end-all troubleshooter.
Read 22 tweets
1/365 - Starting this thread with one of the first Motorcycles to be built in India. The @royalenfield #Bullet350. A Motorcycle which was initially built for the Indian Army but with time found patronage amongst varying genre of people achieving a cult status over the decades. ImageImageImageImage
2/365 - After selling imported Vespas for a while, Bajaj Auto started manufacturing the #Bajaj150 under license in 1959. This was one of the first Scooters in India and certainly changed the 2-Wheeler scene as it was initially available at a significant premium in the market. ImageImageImageImage
3/365 Automobile Products of India (API) was the 1st Scooter maker in India and manufactured the #Lambretta Li 150 Series 2 under license from Innocenti at Bombay in the 1960's. This scooter was very popular and made appearances in a few famous Bollywood movies too. #BikerLife ImageImageImageImage
Read 73 tweets
🚨¡Atención! Bitso participará en el snapshot 📸 de #XRP para la distribución del token #Spark
2/ ☝️Este snapshot será tomado el 11 de diciembre de 2020 a las 18:00 hora de la Ciudad de México, 21:00 hora de Argentina, (00:00 UTC). ⏲️
3/ Es importante que sepas que un snapshot 📸 es cuando se registran los balances de todas las wallets en un momento específico del tiempo. En este caso, se refiere a una “fotografía” de los balances XRP en toda la red de XRP.
Read 10 tweets
So, my thread on how to choose databases for your web apps starts in 20 minutes time. Hope y'all are ready ?However, this time i would love it to be an interactive session where we can learn from each other about database preferences for specific web apps and the use case
#thread
This is what we want to talk about today. Take a really good look at the image below. Study it well. Also notice the words at the 3 vertexes of the triangle. CONSISTENCY, AVAILABILITY & PARTITION-TOLERANCE (Big English - but fear not, we shall demystify all and it'll all be easy)
If we also look at the image in the previous tweet on this thread we can see some popular databases placed at certain sides of the triangle. It is important to note also the positioning of some of a particular point on each side. We shall discuss this in more detail soon. #thread
Read 48 tweets
@AdobeSpark #Feedback. Pinch-in & Pinch-out are not very effective interaction to resize the text boxes precisely in Adobe Spark Android Mobile App. They are good for Zoom-in & Zoom-out from Maps because it don't require precision.
Sometimes when I try to left/right align two text boxes, suppose Title and Content. The auto snap helps me to properly align them but as I lift my finger, the text box slightly displace. That is irritating. I need a tool to fine-tune the movement. This interaction is
for Android Mobile App.

I am not sure that Adobe Spark primarily focusing on Print Media or Social Graphic. As I use Spark mostly for publishing Social Graphic thus I think "Popular" and "Stand out in Social" must be swapped in this page spark.adobe.com/sp/projects.
Read 7 tweets

Related hashtags

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!