1 Any company that tries to build predictive models based only on its own hiring pipeline data is mathematically certain to fail. A short thread -
2 If your company has any momentum at all, you have already hired many people. Those hires came from decisions you made, intentionally or unintentionally, to select a certain group of people. You believe they are "qualified" for your company.
3 "Qualified" means many things. It means the experience on their resume makes you think they can do the job. It also means that when you meet them, your *impression* is that they can do the job. (In @textio terms, it means that they get an interview - something you can measure.)
4 But our decisions about who is/is not qualified are shaped by the people who already work with us. If I see that full-stack engineers from Stanford are doing great on the job, I look for more of them. That is just how people work. As a species, we ❤️ confirmation bias!
5 So a recruiter (whose performance review is based on how well they fill roles, and how many roles they fill) does the normal human thing: They look for people JUST like the successful people already at the company.
6 The hiring manager (whose performance review is based on how well their team's work gets done, which means they need people in role to do the work) does the normal human thing too: They look for people JUST like the successful people already at the company.
7 Over time, your company adopts a set of standards and beliefs about who is "qualified," which determines who even has the chance to get hired. Skills are part of it for almost everyone, yes! But also school, pedigree, gender, race, age, and other stuff. Not all intentional.
8 In this climate, AI based on your own company's hiring data ALONE cannot possibly save you. In fact, all it can do is further doom you. Because your training data set has already encoded all the biases that you've built into your system from those very human hiring decisions.
9 AI that looks ONLY at your own data will be really smart! Smart at helping you make exactly the same decisions you've already made about who you want to hire, and you will make more of those decisions and much faster.
10 An aside, but in the specific case of Amazon, this couldn't possibly have gone anywhere good. This is the company that uses "maniacal" in their job posts 11x more often than the rest of the industry. textio.ai/1000-different…
11 For any AI-based system to work, it MUST incorporate decisions and outcomes from companies that are outside your bias set. A company trying to model stuff on their own is doomed to perpetuate their own biases.
12 On the other hand, a company that's able to use software that aggregates hundreds of millions of data points, from many thousands of companies with thousands of different decision frameworks represented, has a real shot at getting past their own bias.
13 I firmly believe we need software to solve this problem. And I equally firmly believe that a company can't build that software using only their own data set and expect any result different from what they already have.
It's just math.
It's just math.
