Profile picture
, 6 tweets, 3 min read
My Authors
Read all threads
Excited to share a blog post on the connection between #Transformers for NLP and #GraphNeuralNetworks (GNNs or GCNs).

graphdeeplearning.github.io/post/transform… Image
The key idea: Sentences are fully-connected graphs of words, and Transformers are very similar to Graph Attention Networks (GATs) which use multi-head attention to aggregate features from their neighborhood nodes (i.e., words).
cc. @PetarV_93 Image
This connection known to most people, but I've missed having all the information in one place. So I wrote it myself.

I'd love to get feedback and improve this post!
graphdeeplearning.github.io/post/transform…
At our lab, we've already had success taking the Transformer architecture and applying it to graphs, e.g., for graphs of drawings/sketches.

arxiv.org/abs/1912.11258
In Part 2 (coming soon), I will use the @huggingface Transformers library to implement a simple GNN-like Transformer for language modelling and beyond.
@huggingface Discussion thread on @slashML
reddit.com/r/MachineLearn…
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Chaitanya Joshi

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Related threads

Profile picture
Shawn McGuirk 👨🏻‍🔬👨🏻‍💻📝☕🔜🤺💀 ➡️👨🏻‍🎓
@ShawnMcGuirk
My #CSPC2019 live tweeting #THREAD starts here!

I'll reply to this tweet w/ each panel I attend, creating a thread under each w/ live notes

My schedule: 🔗cspc2019.sched.com/shawnmcguirk

And, ICYMI, see the new CSPC magazine for my op-ed about @DSP_SPE & #CapacityBuilding in #scipol!
I wasn't intending on tweeting the symposium, but 2 really good points on skills:

🔹 For communications, the ability to be brief (tweet length summaries!) is key

🔹 Universities should promote organic mentorship by leveraging alumni office, get alums to meet students over ☕!
🔹 Skills development has strong socio-economic dimension. Accessibility to good schooling even as early as Kindergarten can have large impacts later in life. The foundation of transferable (read: critical) skills stays with you as you later develop expertise.

#CSPC2019 #skills
Read 216 tweets
Profile picture
The Influencer League
@influenceleague
Working with Brands: The other day, we shared a little Golden Nugget for you all re: working with brands. Today, we want to drop a few tips & answer some of our most commonly asked questions.
As you all may know, we run influencer campaigns on a daily basis for brands. Therefore, we often receive pitches from influencers. Since we work with influencers every day, we want to share our expert opinion on how to format your pitch email
Working with brands is a two-way street. The relationship you establish with them goes a long way. That initial email is critical and can be the difference between securing a partnership or getting ignored.
Read 19 tweets
Profile picture
Mitchell Wortsman
@Mitchnw
Excited to share our blog on ~Discovering Neural Wirings~ expanding on recent work with Ali Farhadi & @morastegari to appear at #NeurIPS2019 (see thread below for more info)!

Blog: mitchellnw.github.io/blog/2019/dnw/
Preprint: arxiv.org/abs/1906.00586
Code: github.com/allenai/dnw

(1/4)
@morastegari Two cool takeaways:
1) In _some_ ways, the problem of NAS and sparse neural network learning are really two sides of the same coin. As NAS becomes more fine grained, finding a good architecture is akin to finding a sparse subnetwork of the complete graph.

(2/4)
@morastegari 2) It is still possible to play the initialization lottery (see arxiv.org/abs/1803.03635) with a combinatorial number of subnetworks even when the model is sparse.

(3/n)
Read 6 tweets
Profile picture
Meenakshi Prabhune
@minu_pr
It's been 3 years since my first blog. I remember how terrified I was to hit "publish" on my first #blog post. Unleashing my words into the worldwide web was a scary thought. What if people hated my writing? Or worse, what if the article had mistakes? (1/5) #writingtips #SciComm
Importantly, this was supposed to be my first step towards transitioning into a career in #sciencewriting- a field that was a complete black box to my family and friends. Needless to say, everyone, including myself, was skeptical. (2/5)
Anyways, I tried, and learned 2 things over the years. 1) My writing didn't (positively or negatively) break the internet. At it's best and worst, readers moved on with their lives. (3/5)
Read 5 tweets
Profile picture
ramnishkalsi
@ramnishkalsi
Preparing for AWS exams?
Read this thread:
The user can configure SQS, which will decouple the call between the EC2 application and S3. Thus, the application does not keep waiting for S3 to provide the data.
1/n
Auto Scaling enables you to follow the demand curve for your applications closely, reducing the need to manually provision Amazon EC2 capacity in advance.
2/n
Read 554 tweets
Profile picture
Obama_Truth
@ObamaTruth1
1 of 35. #TRUTH #TwitterStorm PLEASE read, #Share, and #Retweet this Very Important thread. Use it on 1/17/18. I will highlight key developments, based on my analysis of #QAnon and ACTUAL events. Many pointers to this, but original source to dig is here: qcodefag.github.io
2. To interpret things:
- I will reference #QAnon material in these posts with {}.
- I will generally place my interpretations OUTSIDE of {} but will use parentheses within to help. Reference web link in post 1 to SEE original Q content.
3. #QAnon is REAL, VERY REAL. There's a massive effort by #DeepState to discredit or label #QAnon as a #LARP.
Q (1/14/18):
{Boards changed due to statements re: private comms - FALSE.
Boards changed due to failure to IDEN accurately.
Boards changed due to MISINFO…
Read 36 tweets

Trending hashtags

#Kodos #share #Cladocopium #StableIsotope #Hawaiian #Sessions #Archaeology #FBI #POTUS #Golem #scicomm #opendata #LeonardNimoy #AAA2018 #SinCity #gayforMoleman #Horizen #poem #SimpsonsFamilyPortrait #HRC #FollowTheWhiteRabbit #Holochain #WeThePeople #Bias #NewAwarenessAwards

Embed code for your website

×
Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!