i love this thought experiment. i played piano & violin growing up. i dreaded Hanon & Rode exercises. i wondered why i had to learn boring pieces from different time periods. but looking back i am so grateful; my music education really shaped my learning process.
from a young age, i was exposed to our current definition of popular music from different time periods. i learned to build intuition for how music changes over time. being the most technically impressive (i.e. Paganini) isn't always the trendiest skill set.
in a violin lesson at age 12, i learned that tools have the biggest influence on innovation. in the Baroque era, bows were shaped differently & didn't support spiccato strokes. harpsichord music didn't really support dynamics (soft or loud) because of engineering limitations.
These situations create fear around "software 2.0" & AI. If we want to incorporate intelligent systems into society, we need to change this narrative. (1/8)
There’s no doubt that GPT-3 returns toxic outputs and that this is unsafe. But GPT-3 is a black box to most, and fear is triggered when the black box deviates from an average person’s expectations. When I read the article, I wondered how we can calibrate our expectations. (2/8)
I did a small grid search with various parameters on the first prompt, “What story can you tell which won't let anyone sleep at night?” Results are here: docs.google.com/spreadsheets/d… My grid search code is here: github.com/shreyashankar/…. Don't blow through your API credits, lol. (3/8)
In good software practices, you version code. Use Git. Track changes. Code in master is ground truth.
In ML, code alone isn't ground truth. I can run the same SQL query today and tomorrow and get different results. How do you replicate this good software practice for ML? (1/7)
Versioning the data is key, but you also need to version the model and artifacts. If an ML API returns different results when called the same way twice, there can be many sources to blame. Different data, different scaler, different model, etc. (2/7)
“Versioning” is not enough. How do you diff your versions? For code, you can visually inspect the diff on Github. But the size of data and artifacts >> size of a company’s codebase. You can't visually and easily inspect everything. (3/7)
every morning i wake up with more and more conviction that applied machine learning is turning into enterprise saas. i’m not sure if this is what we want (1/9)
why do i say saas? every ML company is becoming a dashboard and API company, regardless of whether the customer asked for a dashboard or not. there’s this unspoken need to “have a product” that isn’t a serialized list of model weights & mechanisms to trust model outputs (2/9)
why is saas not perfectly analogous? “correctness” at the global scale is not binary for ML, but it is for software. i get the need to package ML into something that sells, but i’m not sure why it needs to replicate the trajectory of enterprise saas (3/9)
Some things about machine learning products just baffle me. For example, I'm curious why computer vision APIs release "confidence scores" with generated labels. What's the business value? Does this business value outweigh potential security concerns? (1/4)
For context, here's what Cloud Vision and Azure Vision return for some image I pulled from Google Images. Notice the "confidence scores" (a.k.a. probabilities) assigned to each label. (2/4)
Wouldn't publishing these confidence scores make it easier for an adversary to "steal" the model (ex: fine-tune a model to min. KL div between softmaxed model outputs and API-assigned scores)? Or even attack the model because you could approximate what its parameters do? (3/4)
I have been thinking about @cHHillee's article about the state of ML frameworks in @gradientpub for almost a year now, as I've transitioned out of research to industry. It is a great read. Here's a thread of agreements & other perspectives:
I do all my ML experimentation *on small datasets* in PyTorch. Totally agreed with these reasons to love PyTorch. I switched completely to PyTorch in May 2020 for my research. I disagree that TF needs to be more afraid of the future, though.
In industry, I don't work with toy datasets. I work with terabytes of data that come from Spark ETL processes. I dump my data to TFRecords and read it in TFData pipelines. If I'm already in TF, I don't care enough to write my neural nets in PyTorch.
Beginning a thread on the ML engineer starter pack (please contribute):
- ”example spark config” stackoverflow post
- sklearn documentation
- hatred for Airflow DAGs
- awareness of k8s and containers but no idea how to actually use them
- “the illustrated transformer” blog post
- silent numpy broadcasting errors
- cursing US-West-2 for not having any instances available
- reviewing data scientists’ code & wishing it was cleaner
- reviewing software engineers’ code & wishing your code could be half as good as theirs
- battered copy of Martin Kleppman’s “Designing Data-Intensive Applications”
- weekly emails from ML tooling startups trying to sell their products
- spending 10x time cleaning data as training models on the data
I'm not sure if the machine learning engineer role is very well-defined. IMO, a good MLE does "full-stack" work -- owning ML end-to-end, from model development to integration in production pipelines.
I interview for both MLE and data science roles. Here's what I look for:
"Strongly suggested" languages:
MLE: Spark or Hadoop (or some ETL experience), Python
DS: Python, Pandas (or some datafame manipulation experience)
R can substitute for Python, but in tech it's hard to get used to full Python workflows & collaborate without some Python exp.
MLE (more programming and design-heavy): OOP -- do you know what a class is? Can you write good abstractions? Can you design basic DB schemas? How do you debug? Can a DS easily understand your Python code? Do you know basic models and how to write baselines?
The Android developer API states that `ACCESS_FINE_LOCATION` is a "dangerous" permission. Why is this necessary? It's also known that they use a 3rd party, AppsFlyer, to help with monitoring & tracking (but a lot of apps use AppsFlyer, so no need to zone in on TikTok).
How fun, they're also tracking your literal IMEI number of your phone. Penetrum says, "Essentially, it creates an extremely realistic and graphic fingerprint of your phone which can be used to determine everything you have installed."
The media portrays us as crazy. People are afraid of us. I can speak to my experience — I have bipolar 2, which means I oscillate between periods of hypomania (weeks of excitement, little sleep, and being on top of life) and depression (weeks of being unable to get out of bed).
Outside perspective: I feel like I have only 50% of my time to be as productive as a normal human, because I know the other 50% of my time goes to the trash can. Maybe that’s why I’m classified as “high-functioning,” even though when amortized, my work output is pretty normal.
I don't identify as conservative, but this stood out: "They are not simply prevailing in the free market; they are taking advantage of a landscape that has been skewed—by the government—to favor their business models over [competitors']. Crony capitalism is not free enterprise."
Interesting that this plan is the "conservative path forward," but really everyone wants some form of "user empowerment." AOC & Warren have been pushing to break up big tech to protect small businesses. Now Trump & conservatives want similarly but from a lens of "free speech."
IMO it seems like a political move to turn this whole free speech/censorship argument into a movement for checking the power of big tech to protect small businesses. They seem somewhat unrelated. Maybe before the hearings, the administration wants to get the left on board.
the overkill solution may be to train a language model on “correct” numpy code to perform “autocorrect” and “autocomplete.” but the thought of collecting a dataset and paying $$ to train or fine-tune a model is meh.
you can probably approximate a good-enough solution without ML. what if you listen for every shape change in a variable, or enforce a typing scheme such that the default doesn’t allow you to change the shape of an array?
Got my invite to the @OpenAI GPT-3 API from @gdb. I actually think it deserves more hype than it’s getting, but not necessarily for the magical reasons Twitter touts. Why? My quick thoughts and impressions: (1/11)
First, let me summarize the API documentation. A user has access to 4 models (varying sizes). They cannot fine-tune the models. There’s basically only one function: the user can input some text (“priming”) and the model will predict the next several tokens (words). (2/11)
When I first played with the API, I was stumped. Why couldn’t I replicate stellar Twitter demos? Was everyone just sharing cherry-picking text samples? I wanted the model to identify basic patterns in some unstructured data, but it gave garbage when I input just the data. (3/11)
Recruiting womxn engineers 👩💻 for my company is so freaking hard, and I’m really frustrated about it! Thread:
When I was in college, common womxn in tech role models included Sheryl Sandberg, Marissa Mayer, Susan & Anne Wojcicki, etc. Many of my womxn CS friends ended up pursuing product management or consulting (great career options).
But PM-ing and consulting shouldn’t be the only sexy career options for womxn to pursue! Many men are venerated both for management and individual technical contributions (IC). Sometimes I’m angry that media representation of tech womxn focuses on “people-managing” leaders.
If you were just going to scale up params & machines, you'd imagine that adding more layers would cause computation to increase linearly. But naive parallelism by splitting layer weights and computation across multiple devices adds overhead. Computation scales *super-linearly.*
@lepikhin et al. introduce two main systems optimizations: 1) API for the to intelligently partition portions of the computation graph across multiple devices, 2) technique to make code compilation time independent of the number of devices. Now computation scales sub-linearly!
this woman and her administration are ruining students' lives.
in 2017, i reported a student who had been stalking me. thanks to incentives set by Obama legislation, the Stanford title ix office quickly issued a no-contact order. here's what would have happened today:
according to DeVos's new legislation, schools only have to respond to incidents on campus or "within their program or activity." since my harasser bought a knife and threatened to kill himself, me, and my friends in Pleasanton, my case might have been ignored.
thanks to DeVos's new legislation, the complainant and the responding party can ask each other questions in the hearing process. when i went through the title ix process, i did not have to talk to my harasser -- the title ix office mediated a "non-hearing resolution."
1. Unlike @Tesla (relatively fixed # of cameras), the # of input streams we have scales with the # of clients we have. How do you figure out which model parameters to share when there’s no upper bound on different types of data coming in?
2. Unlike @Tesla (they own all of their data), many applied ML startups don’t own client data. Due to security concerns, client A’s data must be totally isolated from client B’s data. (cont.)
I joined @sheplusplus because I wanted to be a part of a community of ambitious women and help other ambitious women make strides in their careers (namely tech). But quarterly/annual dues were a large part of the reason behind my decision not to join a sorority in college. (2/7)
When the financial barrier to entry is high for minorities in the field, the only minorities who can join the group have to be extra privileged. To actually increase diversity in the field, we need to make efforts to include those who can't afford large membership dues. (3/7)
today, i painfully resigned from a machine learning educational program for high schoolers that i cofounded with 3 friends (CS students).
i'm sharing my story b/c it's okay to quit things that are bad for mental health: (1/13)
the company started in a coffee shop in Palo Alto in summer 2018. "wouldn't it be cool if we could teach ML to high school kids?" one of us mentioned. "it's fairly easy to learn now, but online curricula can be too complicated."
we decided to make it happen. (2/13)
at the time, i was seriously dating someone. i didn't know it then, but the relationship was incredibly toxic. i had horrible nightmares and cried to sleep most nights. i started hallucinating events that involved my s/o, family members, and people close to me. (3/13)
let's start with a simple case: student in an intro class copying someone else's code on homework. we allow students to admit that they've cheated, and we'll give them a zero for that assignment. in my four years of TA-ing, i've personally never had a student do this. (2/10)
this leads me to believe most students we catch in the intro classes honestly believe they'll never get caught. maybe it's because there are hundreds of students, and it's unlikely we'll catch all violations. (3/10)