Shreya Shankar Profile picture
Trying to make machine learning work in the real world. Currently 1st ML engineer @viaduct_ai. Previously did ML research @googlebrain & CS @Stanford. She/her.
Eddie Dickey Profile picture 1 added to My Authors
16 Oct
i love this thought experiment. i played piano & violin growing up. i dreaded Hanon & Rode exercises. i wondered why i had to learn boring pieces from different time periods. but looking back i am so grateful; my music education really shaped my learning process.
from a young age, i was exposed to our current definition of popular music from different time periods. i learned to build intuition for how music changes over time. being the most technically impressive (i.e. Paganini) isn't always the trendiest skill set.
in a violin lesson at age 12, i learned that tools have the biggest influence on innovation. in the Baroque era, bows were shaped differently & didn't support spiccato strokes. harpsichord music didn't really support dynamics (soft or loud) because of engineering limitations.
Read 7 tweets
15 Oct
Recently a GPT-3 bot said scary things on Reddit and got taken down. Details by @pbwinston: kmeme.com/2020/10/gpt-3-…

These situations create fear around "software 2.0" & AI. If we want to incorporate intelligent systems into society, we need to change this narrative. (1/8)
There’s no doubt that GPT-3 returns toxic outputs and that this is unsafe. But GPT-3 is a black box to most, and fear is triggered when the black box deviates from an average person’s expectations. When I read the article, I wondered how we can calibrate our expectations. (2/8)
I did a small grid search with various parameters on the first prompt, “What story can you tell which won't let anyone sleep at night?” Results are here: docs.google.com/spreadsheets/d… My grid search code is here: github.com/shreyashankar/…. Don't blow through your API credits, lol. (3/8)
Read 8 tweets
8 Oct
In good software practices, you version code. Use Git. Track changes. Code in master is ground truth.

In ML, code alone isn't ground truth. I can run the same SQL query today and tomorrow and get different results. How do you replicate this good software practice for ML? (1/7)
Versioning the data is key, but you also need to version the model and artifacts. If an ML API returns different results when called the same way twice, there can be many sources to blame. Different data, different scaler, different model, etc. (2/7)
“Versioning” is not enough. How do you diff your versions? For code, you can visually inspect the diff on Github. But the size of data and artifacts >> size of a company’s codebase. You can't visually and easily inspect everything. (3/7)
Read 7 tweets
23 Sep
every morning i wake up with more and more conviction that applied machine learning is turning into enterprise saas. i’m not sure if this is what we want (1/9)
why do i say saas? every ML company is becoming a dashboard and API company, regardless of whether the customer asked for a dashboard or not. there’s this unspoken need to “have a product” that isn’t a serialized list of model weights & mechanisms to trust model outputs (2/9)
why is saas not perfectly analogous? “correctness” at the global scale is not binary for ML, but it is for software. i get the need to package ML into something that sells, but i’m not sure why it needs to replicate the trajectory of enterprise saas (3/9)
Read 9 tweets
20 Sep
Some things about machine learning products just baffle me. For example, I'm curious why computer vision APIs release "confidence scores" with generated labels. What's the business value? Does this business value outweigh potential security concerns? (1/4)
For context, here's what Cloud Vision and Azure Vision return for some image I pulled from Google Images. Notice the "confidence scores" (a.k.a. probabilities) assigned to each label. (2/4) ImageImage
Wouldn't publishing these confidence scores make it easier for an adversary to "steal" the model (ex: fine-tune a model to min. KL div between softmaxed model outputs and API-assigned scores)? Or even attack the model because you could approximate what its parameters do? (3/4)
Read 4 tweets
13 Sep
I have been thinking about @cHHillee's article about the state of ML frameworks in @gradientpub for almost a year now, as I've transitioned out of research to industry. It is a great read. Here's a thread of agreements & other perspectives:

thegradient.pub/state-of-ml-fr…
I do all my ML experimentation *on small datasets* in PyTorch. Totally agreed with these reasons to love PyTorch. I switched completely to PyTorch in May 2020 for my research. I disagree that TF needs to be more afraid of the future, though. Image
In industry, I don't work with toy datasets. I work with terabytes of data that come from Spark ETL processes. I dump my data to TFRecords and read it in TFData pipelines. If I'm already in TF, I don't care enough to write my neural nets in PyTorch.
Read 11 tweets
4 Sep
Beginning a thread on the ML engineer starter pack (please contribute):

- ”example spark config” stackoverflow post
- sklearn documentation
- hatred for Airflow DAGs
- awareness of k8s and containers but no idea how to actually use them
- “the illustrated transformer” blog post
- silent numpy broadcasting errors
- cursing US-West-2 for not having any instances available
- reviewing data scientists’ code & wishing it was cleaner
- reviewing software engineers’ code & wishing your code could be half as good as theirs
- battered copy of Martin Kleppman’s “Designing Data-Intensive Applications”
- weekly emails from ML tooling startups trying to sell their products
- spending 10x time cleaning data as training models on the data
Read 4 tweets
18 Aug
I'm not sure if the machine learning engineer role is very well-defined. IMO, a good MLE does "full-stack" work -- owning ML end-to-end, from model development to integration in production pipelines.

I interview for both MLE and data science roles. Here's what I look for:
"Strongly suggested" languages:

MLE: Spark or Hadoop (or some ETL experience), Python
DS: Python, Pandas (or some datafame manipulation experience)

R can substitute for Python, but in tech it's hard to get used to full Python workflows & collaborate without some Python exp.
Technical concepts:

MLE (more programming and design-heavy): OOP -- do you know what a class is? Can you write good abstractions? Can you design basic DB schemas? How do you debug? Can a DS easily understand your Python code? Do you know basic models and how to write baselines?
Read 11 tweets
3 Aug
Here are some lowlights from my reading of TikTok security reports[1] (for the record, you can learn to avoid these in an intro security or privacy class):

[1] penetrum.com/research
The Android developer API states that `ACCESS_FINE_LOCATION` is a "dangerous" permission. Why is this necessary? It's also known that they use a 3rd party, AppsFlyer, to help with monitoring & tracking (but a lot of apps use AppsFlyer, so no need to zone in on TikTok). Image
How fun, they're also tracking your literal IMEI number of your phone. Penetrum says, "Essentially, it creates an extremely realistic and graphic fingerprint of your phone which can be used to determine everything you have installed." Image
Read 7 tweets
28 Jul
I wrote this while crying in the B40 bathroom at Google because I was going through a depressive episode.

People are talking about bipolar now re: Kanye, but I don’t think there’s enough talk about high-functioning neuroatypicals. Maybe I should speak up, so here goes:
The media portrays us as crazy. People are afraid of us. I can speak to my experience — I have bipolar 2, which means I oscillate between periods of hypomania (weeks of excitement, little sleep, and being on top of life) and depression (weeks of being unable to get out of bed).
Outside perspective: I feel like I have only 50% of my time to be as productive as a normal human, because I know the other 50% of my time goes to the trash can. Maybe that’s why I’m classified as “high-functioning,” even though when amortized, my work output is pretty normal.
Read 17 tweets
27 Jul
I don't identify as conservative, but this stood out: "They are not simply prevailing in the free market; they are taking advantage of a landscape that has been skewed—by the government—to favor their business models over [competitors']. Crony capitalism is not free enterprise."
Interesting that this plan is the "conservative path forward," but really everyone wants some form of "user empowerment." AOC & Warren have been pushing to break up big tech to protect small businesses. Now Trump & conservatives want similarly but from a lens of "free speech."
IMO it seems like a political move to turn this whole free speech/censorship argument into a movement for checking the power of big tech to protect small businesses. They seem somewhat unrelated. Maybe before the hearings, the administration wants to get the left on board.
Read 4 tweets
24 Jul
i’m willing to bet at least O(100) of you have experienced a silent failure because of numpy or broadcasting. even yesterday i found a pandas join bug because i didn’t reset the dataframe’s index. 😢

i‘m wondering how to build basic tools that “catch” such bugs before runtime.
the overkill solution may be to train a language model on “correct” numpy code to perform “autocorrect” and “autocomplete.” but the thought of collecting a dataset and paying $$ to train or fine-tune a model is meh.
you can probably approximate a good-enough solution without ML. what if you listen for every shape change in a variable, or enforce a typing scheme such that the default doesn’t allow you to change the shape of an array?
Read 5 tweets
18 Jul
Got my invite to the @OpenAI GPT-3 API from @gdb. I actually think it deserves more hype than it’s getting, but not necessarily for the magical reasons Twitter touts. Why? My quick thoughts and impressions: (1/11)
First, let me summarize the API documentation. A user has access to 4 models (varying sizes). They cannot fine-tune the models. There’s basically only one function: the user can input some text (“priming”) and the model will predict the next several tokens (words). (2/11)
When I first played with the API, I was stumped. Why couldn’t I replicate stellar Twitter demos? Was everyone just sharing cherry-picking text samples? I wanted the model to identify basic patterns in some unstructured data, but it gave garbage when I input just the data. (3/11)
Read 11 tweets
17 Jul
Recruiting womxn engineers 👩‍💻 for my company is so freaking hard, and I’m really frustrated about it! Thread:
When I was in college, common womxn in tech role models included Sheryl Sandberg, Marissa Mayer, Susan & Anne Wojcicki, etc. Many of my womxn CS friends ended up pursuing product management or consulting (great career options).
But PM-ing and consulting shouldn’t be the only sexy career options for womxn to pursue! Many men are venerated both for management and individual technical contributions (IC). Sometimes I’m angry that media representation of tech womxn focuses on “people-managing” leaders.
Read 8 tweets
3 Jul
Dear ML community, read past the abstract! Before you jadedly scream "why train a gajillion parameters on a gajillion TPUs," realize that @lepikhin et al. did so much more. Thread:
If you were just going to scale up params & machines, you'd imagine that adding more layers would cause computation to increase linearly. But naive parallelism by splitting layer weights and computation across multiple devices adds overhead. Computation scales *super-linearly.*
@lepikhin et al. introduce two main systems optimizations: 1) API for the to intelligently partition portions of the computation graph across multiple devices, 2) technique to make code compilation time independent of the number of devices. Now computation scales sub-linearly!
Read 6 tweets
7 May
this woman and her administration are ruining students' lives.

in 2017, i reported a student who had been stalking me. thanks to incentives set by Obama legislation, the Stanford title ix office quickly issued a no-contact order. here's what would have happened today:
according to DeVos's new legislation, schools only have to respond to incidents on campus or "within their program or activity." since my harasser bought a knife and threatened to kill himself, me, and my friends in Pleasanton, my case might have been ignored.
thanks to DeVos's new legislation, the complainant and the responding party can ask each other questions in the hearing process. when i went through the title ix process, i did not have to talk to my harasser -- the title ix office mediated a "non-hearing resolution."
Read 6 tweets
4 May
in last night’s dream my college friends and I showed up late to 7th grade Texas History and we did NOT complete bellwork in time, which was to write BFS in C++ to work on a large graph
my naive implementation was too slow, so i parallelized it. then it broke, so i added ‘mov rax, 1’ for a fake “fix” just as the bell rang :(
technically i passed all the test cases, which goes to show you should really include some cases in your unit tests that you know will fail
Read 4 tweets
26 Apr
I 100% recommend this @karpathy talk about multi-task learning at scale. @ericjang11 covers some of the main points well.

But as the 1st ML engineer at a startup that sells an ML platform to automakers, I want to talk about other problems that many applied ML startups face:
1. Unlike @Tesla (relatively fixed # of cameras), the # of input streams we have scales with the # of clients we have. How do you figure out which model parameters to share when there’s no upper bound on different types of data coming in?
2. Unlike @Tesla (they own all of their data), many applied ML startups don’t own client data. Due to security concerns, client A’s data must be totally isolated from client B’s data. (cont.)
Read 15 tweets
22 Jan
Hot take: @the_wing is an adult sorority ($200+ monthly dues, tight community, philanthropy efforts)

I love the idea of empowering women, but I think it's super important to do so while being inclusive to all identifying women. Thread (1/7):

fortune.com/2020/01/22/wew…
I joined @sheplusplus because I wanted to be a part of a community of ambitious women and help other ambitious women make strides in their careers (namely tech). But quarterly/annual dues were a large part of the reason behind my decision not to join a sorority in college. (2/7)
When the financial barrier to entry is high for minorities in the field, the only minorities who can join the group have to be extra privileged. To actually increase diversity in the field, we need to make efforts to include those who can't afford large membership dues. (3/7)
Read 7 tweets
21 Dec 19
cw: mental illness, hallucinations

today, i painfully resigned from a machine learning educational program for high schoolers that i cofounded with 3 friends (CS students).

i'm sharing my story b/c it's okay to quit things that are bad for mental health: (1/13)
the company started in a coffee shop in Palo Alto in summer 2018. "wouldn't it be cool if we could teach ML to high school kids?" one of us mentioned. "it's fairly easy to learn now, but online curricula can be too complicated."

we decided to make it happen. (2/13)
at the time, i was seriously dating someone. i didn't know it then, but the relationship was incredibly toxic. i had horrible nightmares and cried to sleep most nights. i started hallucinating events that involved my s/o, family members, and people close to me. (3/13)
Read 13 tweets
14 Oct 19
i bet this isn't the only plagiarizing paper on arxiv. @sirajraval is getting called out partially because he is an influencer, and we don't want fraudulent influencers in ML.

as a TA, i've seen many plagiarism cases. why do people commit academic dishonesty? my thoughts: (1/10)
let's start with a simple case: student in an intro class copying someone else's code on homework. we allow students to admit that they've cheated, and we'll give them a zero for that assignment. in my four years of TA-ing, i've personally never had a student do this. (2/10)
this leads me to believe most students we catch in the intro classes honestly believe they'll never get caught. maybe it's because there are hundreds of students, and it's unlikely we'll catch all violations. (3/10)
Read 10 tweets