gcc startup.c -o ./startup. Focused on machine learning & society. Previously @Salesforce Research via @MetaMindIO. @Harvard '14, @Sydney_Uni '11. 🇦🇺 in SF.
Jun 21, 2022 • 7 tweets • 3 min read
To add to a night of technical oddities there are three Cruise vehicles, all (literally) driverless, stuck at and partially blocking the corner of Geary and Mason 😅
There were originally four Cruise vehicles but one eventually made a grand escape. The leading Cruise vehicle has been there at least fifteen minutes as that's how long I had to wait for fast food. Occasionally one of them would lurch forward a little just for added suspense 🙃
Nov 7, 2019 • 8 tweets • 2 min read
For those in the language modeling space, a question regarding perplexity as a metric with varying tokenization:
- Is there a hard proof showing for a dataset D being tokenized using A and B that the perplexity is equivalent?
- Does that proof take into account teacher forcing?
I ask as I have never seen a proof and always assumed smarter people than myself had thought about it. Intuitively I felt it reasonable until I recently began pondering over the teacher forcing aspect which is essentially giving your model supervision, including at test time.
Nov 6, 2019 • 7 tweets • 3 min read
In Dec 2016 Uber started taking _paid rides_ in _self driving cars_ without even filing for an autonomous testing permit in CA. That first day in SF it blew multiple red lights and had disengagement numbers hundreds of times worse than other companies.
Less than two years later, Uber having upped and left San Francisco due to their egregious behaviour, their self driving car killed someone. I collected why, in a thread, I had zero faith in their ability to safely execute and their checkered past.
Deep learning training tip that I realized I do but never learned from anyone - when tweaking your model for improving gradient flow / speed to converge, keep the exact same random seed (hyperparameters and weight initializations) and only modify the model interactions.
- Your model runs will have the exact same perplexity spikes (hits confusing data at the same time)
- You can compare timestamp / batch results in early training as a pseudo-estimate of convergence
- Improved gradient flow visibly helps the same init do better
Sep 1, 2019 • 13 tweets • 5 min read
I'm incredibly proud that the low compute / low resource AWD-LSTM and QRNN that I helped develop at @SFResearch live on as first class architectures in the @fastdotai community :)
I think the community has become blind in the BERT / Attention Is All You Need era. If you think a singular architecture is the best, for whatever metric you're focused on, remind yourself of the recent history of model architecture evolution.
Aug 21, 2019 • 9 tweets • 4 min read
A brilliant article with insights from @emilymbender, @sarahbmyers (@AINowInstitute), and more. But taking a step back:
As an NLP researcher, I'm asking what the freaking hell is anyone doing grading student essays with automated tools that I'd not trust on my academic datasets?
In 18 states "only a small percentage of students’ essays ... will be randomly selected for a human grader to double check the machine’s work".
In writing you're tasked with speaking to and convincing an audience through a complex, lossy, and fluid medium: language.
Jul 22, 2019 • 8 tweets • 2 min read
What is OpenAI? I don't know anymore.
A non-profit that leveraged good will whilst silently giving out equity for years prepping a shift to for-profit that is now seeking to license closed tech through a third party by segmenting tech under a banner of pre/post "AGI" technology?
The non-profit/for-profit/investor partnership is held together by a set of legal documents that are entirely novel (=bad term in legal docs), are non-public + unclear, have no case precedence, yet promise to wed operation to a vague (and already re-interpreted) OpenAI Charter.
Apr 26, 2019 • 11 tweets • 3 min read
This is such an extreme outlier I'm ready to believe that a nation state actor may have an expert AI team* reverse engineering and gaming social graphs and recommendation algorithms at scale silently.
* Asymmetric black hat research (see below) and/or StuxNet level competent 1/N
It's quite possible for machine learning to have exploits as fundamentally severe and retrospectively obvious as the NSA's 13+ year head start in differential cryptography. White hat research is a terrible proxy for black hat research - especially for AI.
@Miles_Brundage@jackclarkSF@OpenAI@NvidiaAI (a) Released after I'd already spoken to the journalist
(b) Fragmented information that's important to the narrative
(c) A week of 256 TPUv3 cores which means it's trivially reproducible by larger companies especially with code ..? What's the responsible disclosure narrative?
@Miles_Brundage@jackclarkSF@OpenAI@NvidiaAI I had literally not even seen the week training estimate yet it was in reply to a post that uses my tweet as a source for the 256 TPUv3 core number (via @katyanna_q's article). I'm at the center of this web and I personally have every and no clue as to what's going on.
Feb 18, 2019 • 5 tweets • 6 min read
@jackclarkSF@OpenAI@Miles_Brundage Preface: I love the team at @OpenAI, I hope that's clear and obvious.
My main issue: way more than once @OpenAI have been the source of AI hyped narratives, intentional or not. The responsibility falls on the AI community to try to hold off the avalanche of misinformation.
@jackclarkSF@OpenAI@Miles_Brundage My feelings are almost exactly the same as two years ago from the @OpenAI DotA 1v1 narrative and I'm disappointed as I thought we'd moved beyond that.
Today's meta-Twitter summary for machine learning:
None of us have any consensus on what we're doing when it comes to responsible disclosure, dual use, or how to interact with the media.
This should be concerning for us all, in and out of the field.
I know each person and organization and they're all well intentioned. Some are getting aggravated at each other but I believe that's as they all see this as important and are reacting to a live-fire exercise split across the already contentious areas of {ML, PR, AI hype, ...}.
Sep 15, 2018 • 6 tweets • 3 min read
I wasn't going to do @Twitter's "Tweet the meaning behind your Twitter handle" - but then I remembered many people have introduced me with the wrong first name when they can't use @Smerity (including some _very_ high profile introductions 😅), so:
If you're someone lucky enough to have introduced me incorrectly, I promise I don't mind, it's just hilarious ^_^ You're always welcome to use Smerity or to introduce me as Smerity - and if it's a really important intro maybe just copy+paste ;)
Sep 3, 2018 • 5 tweets • 2 min read
Digitization is vital to preservation as bits are our best insurance against time and entropy. Bits are resilient to chronic underfunding as storage gets cheaper. Bits can never replace the original but can allow universal access for sharing our culture and preserving it.
Underfunding may be the tragedy here but there are many tragedies no amount of funding can prevent over a long enough timeframe. Digitization may only offer a ghost - but it's a ghost that will never age and can and will get captured in the fabric of our infinite digital museum.
Aug 24, 2018 • 11 tweets • 6 min read
Congratulations to the @OpenAI Five team! Whilst the bots may have lost we've all gained from watching your journey :) My live stream commentary is linked and below I'll add a bit of post game commentary. twitch.tv/videos/3011031…
First, we should all note how impressive it has been to watch the restrictions removed in the last two or so months. The largest was from a fixed set of heroes to a small hero pool but there are many on top of that. IMHO the game is now functionally equivalent to normal gameplay.
Aug 22, 2018 • 4 tweets • 3 min read
I'm live streaming from now through today's @OpenAI Five match. The match right now will be human vs human and I'll use it to explain some of the broader concepts and to get used to the live commentary myself. I'll ping again as the @OpenAI match begins! twitch.tv/smerity
The match will start soon - I'm streaming live and provide insight as to how the bots are playing and what they're doing! :) Again, no DotA experience. If you're an experienced DotA player, you'll enjoy how I describe this for non DotA players! ^_^
Stream starting in a few minutes where I'll pull apart a few minutes of @OpenAI bot vs bot play. No past DotA experience necessary! I'll record and post online after too :) twitch.tv/smerity/
Highlights:
Sven (blue) + Lion (red) gank (=surprise attack) an enemy
Techniques: blink (=short range teleport) to initiate and then chain stuns (i.e. if you and your team mate both have a 2 second stun it's best to stun A, wait 2 seconds, stun B, ...) twitch.tv/videos/3003208…
Aug 21, 2018 • 6 tweets • 4 min read
The way this is written it sounds as if these were recently revealed details of @OpenAI Five, pulled out of secrecy. The post which clarified API use (and debate around it) was literally a year ago (1v1 at TI 2017) and @OpenAI have been impressively transparent since.
I complained thoroughly about it at the time. The main issue was that none of the technical details were released with the surprise reveal at TI 2017 which, in combination with @elonmusk overhyping the result in media, lead to speculation then rollback.
A sad secret that most professional conference goers know: talks can sometimes be the absolute worst use of your time. The smartest people in your field from all around the world are now physically in the same room as you - and then you all sit silently.
My favourite conference experience ever was an accident. What started as a few people catching up for lunch became an impromptu language modeling / translation / knowledge graph discussion which literally became a round table (we found a round black marble table + sat at it) ^_^
Aug 17, 2018 • 25 tweets • 8 min read
I've been meaning to write about this for some time but I rarely write atm and this is a depressing topic. Anger helps me write however and boy am I angry. Sooooo, I know some want to reclaim/recover r/MachineLearning (r/ML) but I don't think that's even feasible. Tweetstorm:
Underlying broader @reddit community problems for r/ML (and other r/...):
- Anonymity allows incivility
- Brigades can easily "launch" from other subreddits (i.e. "We hate women" subreddits)
- r/Futurism and other "science (fiction)" subreddits leak in meaning results are hyped
Aug 10, 2018 • 8 tweets • 3 min read
A friend interested in growing plants read about the issues with fake seeds (plastic, grass not <plant>, ...) and pondered "Imagine a post apocalyptic world where seeds are guarded in seed libraries?". Enter: "The Windup Girl".
Spoiler-free "setting the scene": Genetic engineering lead to food wars where companies created plagues and pests to destroy natural and competitor edible plants in order to lock in a monopoly. Think terminator / suicide seeds but next level. en.wikipedia.org/wiki/Genetic_u…
Aug 7, 2018 • 4 tweets • 3 min read
$TSLA went from a company where I genuinely championed their vision years ago to a dumpster fire where @elonmusk takes time away from trolling and harassing on @Twitter to (at best) break securities law on @Twitter ...
"Investors can't tell if Elon Musk was making a weed joke or if he’s actually talking about investments" should be a headline from @TheOnion, not a necessary topic of financial discussion for a company that already has a laundry list of serious issues.
