Thread by @WellsLucasSanto

Being shown now at #FAccT22! The production quality on this paper video is so good 🥺 A must-watch for those interested in this growing area of AI auditing

https://twitter.com/Borhane_B_H/status/1537501359410405380

Video is also moving at light speed 😅 so I haven't been able to take notes, catch the video, and tweet about it.

Some highlights:
* Client confidentiality prevents second-party auditors from sharing data about the audits
* Auditing regulation: ...there are few
#FAccT22

Here is the full video from @AJLUnited on "Who Audits the Auditors?" which is a must-watch from #FAccT22. (There is an abridged 4 minute version you can view as well!)

From @schock in #FAccT22 chat:

- Majority of auditors want regulation that requires disclosure of results
- Majority did *not* think it was critical to check if there was harm reporting
- Only two respondents provided evidence that impact communities are involved in audit!

Another take-away: first-party auditors (those internal to the companies) *want* to share summaries of audit results and their methods, but they're not allowed to (due to NDA or other company restrictions). #FAccT22

Slide on AI auditing. AI auditing checks whether systems meet expectations in these various areas, including regulatory compliance, consent, labor practices, bias, transparency, effectiveness, energy use, security, and vulnerable community impacts. #FAccT22

I've got the feeling that we'll be referring to this terminology for years to come! Some defining work here.

First-party auditors: internal to the company
Second-party auditors: contractors/consultants
Third-party auditors: independent researchers, journalists

#FAccT22

First-party auditors: have high levels of access, but low levels of transparency

Second-party auditors: questions about whether they can disclose & accountability incentives

Third-party auditors: limited/no access to target systems

#FAccT22

One of the *big* topics of discussion around auditing in past years: Cathy O'Neil (second-party) mentions that she can't demand disclosure, or else she won't be hired.

82% of auditors think that public disclosure of audit results *should* be mandated by regulation. #FAccT22

One of my biggest takeaways from this #FAccT22 session:

Respondents agreed that "Companies do not take action on ethical AI issues unless they face public pressure."

"The best way to make sure that an issue is attended to is to have the press report on it."