Thread by @jennifercobbe

FAccT paper with @mikarv and @jatinternet - on accountability in algorithmic supply chains

We explore the dynamics of AI supply chains, their implications for accountability, and why we need to understand their legal and political economic context

Link: papers.ssrn.com/sol3/papers.cf…

A lot of algorithmic accountability work is based (implicitly or otherwise) in trying to understand the actions of a single organisation which makes and deploys an AI system

These tools are useful but this focus is a mistake

Digital technologies are now often built around cloud-based services, using a client-server model where compute is centralised but deployment of tech is distributed, producing complex supply chains driven by data flows between multiple actors

These supply chains are dynamic, transient, and unstable - they may come into existence when executed and disappear when they complete, the actors involve might differ each time a chain is instantiated, and they change over longer periods as services and applications develop

Data-driven supply chains are now key infrastructures in political economy of informational capitalism - producing and shaping data flows, enabling functionality, and centralising power with core players who can extend it over others - and increasingly underpin AI technologies

Technological and economic barriers to entry for advanced AI tech mean many general purpose and generative systems will in future be produced or controlled by a few key players - primarily Amazon, Microsoft, and Google - and distributed as services running on their infrastructure

Even where those companies don't build systems themselves, other providers of AI services like Open AI use their cloud infrastructure for production and distribution (e.g. Open AI uses MS Azure for its cloud infrastructure and for distribution of its API-based services)

The big players are now highly integrated both horizontally (across markets and sectors) and vertically (across production and distribution of AI systems), with significant technical expertise and resources, and benefitting from economies of scale and data from many customers

So interdependencies and balances of power in algorithmic supply chains are often highly asymmetric - allowing big providers to leverage APIs and standard form terms of service to structure their relations with customers and extend control over deployment and use of their tech

AI providers also extend their own supply chains across borders for regulatory arbitrage - outsourcing things like dataset production to companies in jurisdictions with weaker data protection and privacy laws, lower wages, and fewer workers' rights and employment protections

Compute is centralising around big providers, but algorithmic supply chains exhibit *distributed responsibility* - a split between production activities (by providers and their suppliers), deployment (by customers), and potentially use (by end-users) of AI systems as services

Existing and proposed laws like GDPR and even the AI Act don't properly reflect this distribution of practical responsibilities and will struggle to contend with what's happening here. This is not good! We need to fix these laws to rein in power and protect people and societies

Algorithmic supply chains also have an *accountability horizon* - it's very difficult for any one actor to 'see' more than one or two steps upstream or downstream from their position in the chain, and all but impossible for others to understand what's happening within chains

This makes it hard for providers to properly frame problems when developing systems - they're less able to account for the many contexts and use-cases of customers' applications

It's also difficult for customers to know which systems and services are appropriate to their needs

It also makes risk management reqs in laws like GDPR and the AI Act hard for anyone to meet - providers will struggle to foresee risks in the many and varied applications and contexts where customers deploy their systems - even one step down the chain, let alone further away

These problems are exacerbated by the scale services operate at - even if it was easy to see across chains, providers have so many customers and general purpose AI systems have so many potential applications and contexts of use that it's practically impossible to manage

All of these things challenge existing governance and accountability mechanisms and make reliably allocating accountability to the appropriate actor difficult

Yet we have few legal, technical, or other ways of understanding algorithmic supply chains and addressing these problems

If algorithmic accountability is to be a mechanism for challenging power - as it should be - then we need much more work to understand the dynamics of supply chains, to build ways of investigating and contesting them, and to reliably allocate accountability across them