I 100% recommend this @karpathy talk about multi-task learning at scale. @ericjang11 covers some of the main points well.
But as the 1st ML engineer at a startup that sells an ML platform to automakers, I want to talk about other problems that many applied ML startups face:
But as the 1st ML engineer at a startup that sells an ML platform to automakers, I want to talk about other problems that many applied ML startups face:
1. Unlike @Tesla (relatively fixed # of cameras), the # of input streams we have scales with the # of clients we have. How do you figure out which model parameters to share when there’s no upper bound on different types of data coming in?
2. Unlike @Tesla (they own all of their data), many applied ML startups don’t own client data. Due to security concerns, client A’s data must be totally isolated from client B’s data. (cont.)
2 (cont). But given that we don’t want to reinvent the wheel for each client, and multi-task learning could improve performance for individual tasks, where in the pipeline do we share code amongst different data lakes?
3. Unlike @Tesla (where the customers don’t need to understand ML), many applied ML startups need to communicate ML ideas to their clients. For example, if one of our clients has super imbalanced data (few positively labeled examples), I need to communicate that accuracy (cont.)
3. (cont.) isn’t a good metric. I need to explain good science to the client — we should have multiple validation sets, we need to isolate test data, etc — because we need to demonstrate lots of value in very little time.
4. This wasn’t mentioned in the talk, but a BIG problem applied ML startups face is that clients expect constant engagement and improvements in metrics. We meet with clients weekly, but it’s impossible to have significant model improvements every week! How do you separate (cont.)
4. (cont.) short-term gains from long-term investments and ensure that both are happening in parallel, when you inevitably don’t have enough data scientists or engineers?
5. To add to the team dynamics complexity issues, data scientists at applied ML startups can be assigned to a task (ex: predicting engine failure) and a client. Lots of people are fighting to have shared resources to improve their individual task performances. If I (cont.)
5. (cont.) introduce changes that improve task P’s performance and slightly change parameters in a model applied to task Q's data, how do I quantify the performance change? Should my changes be merged to master? I need to inform the data scientist assigned to task B, right?
6. A tool I would love to implement that draws from my systems background: “semaphores” to limit access to model parameters. We’re essentially trying to solve the readers-writers problem (Courtois et al.) for ML, where “writing” in this concept is “(re)training a model.” (cont.)
6. (cont.) In the extreme case, only one data scientist can write/train a model at a time (while having multiple readers). But this makes development too slow, so you could limit the number of writers (essentially lowering the probability of a “merge conflict”).
7. Another tool I would love to implement: a “diff checker” for models. Which parameters have been significantly altered, so I can alert the code owners for that section of the model? How do you determine what “significantly altered means? How do you create a suite (cont.)
7. (cont.) of unit tests to identify which tasks have improved or decreased performances when a model is updated? I’d draw from ideas established in Eraser (Savage et al.), a tool that detects race conditions and prioritizes false positives over false negatives.
8. To summarize, there is a HUGE gap between ML research and applied ML. This list of issues is by no means exhaustive of the challenges we face day-to-day. This thread wouldn’t be complete without a plug for my company — if these challenges excite you, join us at @viaduct_ai !
