Thread by @geomblog

The 5 principles in the #aibillofrights are common sense: systems should work, they shouldn't discriminate, they shouldn't use data indiscriminately, they should be visible and easy to understand, and they shouldn't eliminate human interlocutors. whitehouse.gov/ostp/news-upda…
🧵
1/n

But there's a lot going on "under the hood". Let's unpack them one by one.

Today: Safe and Effective Systems

The principle is clear: Rights-impacting systems should work, and work well. But what does that really mean? Don't the systems we deploy work already? 2/n

Sadly, no. A great paper by @rajiinio, @ziebrah,@Aaron_Horowitz , and @aselbst called "The Fallacy of AI Functionality" unpacks all the ways in which automated systems claim to work but don't. dl.acm.org/doi/10.1145/35…
3/n

@random_walker has written about what he calls "AI Snake Oil" (cs.princeton.edu/~arvindn/talks…) and indeed is now writing a book about it with @sayashk (aisnakeoil.substack.com)
4/n

So automated systems don't always work. And when they fail, there are many consequences, some of which are listed in the "Why this principle is important" section in the Technical Companion to the #AIBillofRights 5/n

So what should we do about it? Because there are many ways in which automated systems can fail to work, or work well, there are many different checks that are needed to ensure that they are safe when deployed. 6/n

A system that isn't designed keeping in mind the needs of the people it will impact will likely not serve the community's needs: so we need to make sure systems are designed with public consultation and input (like we take input for other key decisions)
7/n

Systems should work and continue to work once deployed; and should be rigorously evaluated for different kinds of risks (e.g with the @NIST AI RMF (nist.gov/itl/ai-risk-ma…)) regularly. There MUST be the option of taking the system down and not deploying it altogether. 8/n

Not all harms are foreseeable, granted. But many are. And sometimes it just takes that extra effort to think through potential harms. That effort is worthwhile. 9/n

AI systems in particular feed on data. But is the data the right data? is it just compounding mistakes from other systems that feed into this one? Have you made sure that you're using relevant data that has been established as pertinent to the task at hand?
10/n

How sensitive is the domain that you're working with? If it's very sensitive, you should take extra precautions with the data you use and make sure you really understand where it comes from. We don't have the luxury of just grabbing whatever data comes in handy. 11/n

All of this needs supervision -- aka governance. there have to be processes and decision-making to evaluate the results of testing. I hear all the groans: "you're trapping innovation and slowing it down". If people are being affected, then it's warranted. Speed... kills. 12/n

We have ways of doing this. For e.g. @datasociety put out a very nice guideline for evaluating impact assessments (datasociety.net/library/assemb…) and the GAO had a great report last year (that @kdphd helped co-author before going to @WHOSTP) gao.gov/products/gao-2… 13/n

And it needs independent testing and reporting. It's hard to get excited about transparency requirements. But without it, there's no incentive to actually do a good job on testing and evaluation, and without independent testing there's no incentive to do it right. 14/n

Hopefully this illustrates the key design elements for ensuring that rights-impacting systems are safe and effective. Next up: Algorithmic Discrimination. 15/n