Thread by @krishnagade

Zillow has recently shut down its AI-enabled iBuyer program that overpaid thousands of houses in summer 2021, along with laying off 25% of its staff.

Are teams managing these risks w/ AI well?

THREAD: How to build a robust Model Risk Management (MRM) process in your company?

1/ Zillow invested a ton of money into its AI-enabled home-flipping program called Zillow Offers.

They bought up thousands of houses per month, whereupon the homes would be renovated and sold for a profit.

Unfortunately, things didn’t go to plan.

bloomberg.com/news/articles/…

2/ Zillow's CEO recently said that they failed to predict house price appreciation accurately:

“We’ve determined the unpredictability in forecasting home prices far exceeds what we anticipated.”
wsj.com/articles/zillo…

3/ As @galenward tweeted a few days ago, it is unclear where in the ML stack Zillow’s failure lives:

We can only hypothesize what could have happened in Zillow’s case and propose how Model Risk Management could have helped here.

https://twitter.com/galenward/status/1455680292438437892?s=20

4/ A problem with modeling asset-valued entities, is that they depreciate based on human usage. Let’s take 2identical houses in the same zip code, each is 2,100 square feet and has the same number of bedrooms and bathrooms. How do we know one was better maintained than the other?

5/ Additionally there are global market conditions such as interest rates, GDP, unemployment, and supply and demand in the market that could affect home prices. 

Essentially, there are always things that impact a home price that can’t be easily measured and included in a model.

6/ Another thing to worry about is the non-stationarity nature of data. House prices change over time happen due to house depreciation as well as market conditions. 

So, teams look at cross-sectional data on house-specific variables such as square footage, year built, or zip.

7/ This is where the Model Risk comes in, because implicit in the machine learning process of dataset construction, model training, and model evaluation is the assumption that the future will be the same as the past. 

Data scientists call this Data Drift.

8/ In effect, ML algorithms search through the past for patterns that might generalize to the future. But the future is subject to constant change, and production models can deteriorate in accuracy over time due to data drift.

9/ One or more of these types of data drift could have caused Zillow’s models to deteriorate in production.

Therefore Model performance monitoring becomes critical to catch issues emerging due to the non-stationary nature of the data proactively.

10/ At a bare minimum, teams need to monitor the following things.

a) Performance metrics like MSE, MAPE, etc.
b) Shifts in feature distributions over time.
c) Quality of data that is coming in in the production pipelines.
d) Finally alerts that can fire when things change.

11/ Another thing that could have happened in Zillow is If there was a human misinterpretation or misuse of AI.

Explainability helps provide more information and intuition about how a model operates, and reduce the uncertainty that it will be misused.

12/ XAI enables us to ask questions like:

a) Given a model trained mostly on older homes ranging from 100K-300K, can I trust a model’s prediction that a newly built house costs $400,000?
b) Given 5 houses in a neighborhood, what distinguishes them and their prices?
c) ...

13/ Model risk occurs primarily for two reasons: (1) a model may have fundamental errors and produce inaccurate outputs for the intended business uses; (2) a model may be used incorrectly or inappropriately or there may be a misunderstanding about its limitations and assumptions.

14/ We really don’t know about Zillow’s model risk process, it is possible they look into these things.

One thing is clear: It is imperative for organizations to establish a strong risk culture around how their AI is developed, deployed, and operated.

blog.fiddler.ai/2021/12/zillow…