🤔When an algorithm causes harm, is discontinuing it enough to address its harms?
💡Our #FAccT2022 paper introduces the concept of the _The Algorithmic Imprint_ to show how algorithmic harms can persist long after it’s discontinued
📜arxiv.org/abs/2206.03275…
A tweetorial👇
1/n
💡Our #FAccT2022 paper introduces the concept of the _The Algorithmic Imprint_ to show how algorithmic harms can persist long after it’s discontinued
📜arxiv.org/abs/2206.03275…
A tweetorial👇
1/n
🎯 We situate the concept of the Algorithmic Imprint through the 2020 Ofqual grading controversy where an algorithm initially graded the GCE exams
🤯 Despite “removal” of the algorithm, its effects persisted in the algorithm’s “afterlife”
🎁 What are the implications?
2/n
🤯 Despite “removal” of the algorithm, its effects persisted in the algorithm’s “afterlife”
🎁 What are the implications?
2/n
Implications for algorithm deployers:
🎯Critically ask: does the problem really *need* an algorithmic intervention?
💯Despite being made out of “soft”ware, algorithms can leave hard imprints on society
💡That is, there’s no simple “undo” button for algorithmic deployments
3/n
🎯Critically ask: does the problem really *need* an algorithmic intervention?
💯Despite being made out of “soft”ware, algorithms can leave hard imprints on society
💡That is, there’s no simple “undo” button for algorithmic deployments
3/n
Implications for algorithmic impact assessments:
🎯 We need to expand how we think about algorithmic impact
💯 An imprint-aware mindset extends the scope of analysis
💡 It allows us to "see" effects that would otherwise be less visible in the algorithm’s “afterlife”
4/n
🎯 We need to expand how we think about algorithmic impact
💯 An imprint-aware mindset extends the scope of analysis
💡 It allows us to "see" effects that would otherwise be less visible in the algorithm’s “afterlife”
4/n
Implications for designers:
🗺️Imprint-awareness can allow us to map how upstream requirements (eg. data requirements) can cause harmful downstream effects (eg. unfair outcomes)
🚀We can use HCI methods like scenario-based & value-sensitive design to address these
5/n
🗺️Imprint-awareness can allow us to map how upstream requirements (eg. data requirements) can cause harmful downstream effects (eg. unfair outcomes)
🚀We can use HCI methods like scenario-based & value-sensitive design to address these
5/n
🎯Our paper focuses on Bangladesh–not the UK–for the Ofqual scandal
🧐Did you know that GCE exams are taken in 130+ countries?
🎁We present the first coherent timeline that chronicles the Bangladeshi side of the Ofqual scandal & view it through the lenses of the imprint
6/n
🧐Did you know that GCE exams are taken in 130+ countries?
🎁We present the first coherent timeline that chronicles the Bangladeshi side of the Ofqual scandal & view it through the lenses of the imprint
6/n
🇧🇩Why Bangladesh?
🎯Despite commonwealth countries’ substantial market share of GCE exams, they are severely under-explored for the Ofqual scandal
🎯Investigate how algorithms made in the Global North impacts stakeholders in the Global South with less voice in the process
7/n
🎯Despite commonwealth countries’ substantial market share of GCE exams, they are severely under-explored for the Ofqual scandal
🎯Investigate how algorithms made in the Global North impacts stakeholders in the Global South with less voice in the process
7/n
💫Returning to the events, on the surface, things look good–the boards dropped the algorithmic component, implying reversal of its effects.
🧐Was that really the case–were the algorithmic effects undone?
🚀This is where we introduce the algorithmic imprint to answer the Q
8/n
🧐Was that really the case–were the algorithmic effects undone?
🚀This is where we introduce the algorithmic imprint to answer the Q
8/n
A metaphor can be useful here:
💡 Algorithmic imprints are much like the remnant traces of a palimpsest–scrolls where old text is erased to make room for new text, but its traces remain
🔥 Just like traces in a palimpsest, algorithms leave hard imprints on society
9/n
💡 Algorithmic imprints are much like the remnant traces of a palimpsest–scrolls where old text is erased to make room for new text, but its traces remain
🔥 Just like traces in a palimpsest, algorithms leave hard imprints on society
9/n
📍To understand the imprint in an applied setting, we focus on the Bangladeshi events capturing >1 year-long community engagement
🗺️We’ve stitched a timeline through 47 interviews
🎯To help you track this, we will break this down into 4 acts spanning Mar-Aug, 2020
10/n
🗺️We’ve stitched a timeline through 47 interviews
🎯To help you track this, we will break this down into 4 acts spanning Mar-Aug, 2020
10/n
Act 1:
▶️ Mar 20, 2020 exams get canceled but only in the UK, leaving BD students stranded
▶️ Following global outcry, on Mar 23, exams are canceled globally
💯 These are important pre-conditions that make the algorithm possible & its resulting imprint
11/n
▶️ Mar 20, 2020 exams get canceled but only in the UK, leaving BD students stranded
▶️ Following global outcry, on Mar 23, exams are canceled globally
💯 These are important pre-conditions that make the algorithm possible & its resulting imprint
11/n
Act 2 is action-packed, running from Apr 1–Aug 10, 2020:
🎯The central topic: how alternative assessment was done in lieu of actual exams
The assessment has 3 components–
1. Teacher Assessed Grades (TAGs)
2. Ranking
3. Algorithmic Standardization
Let’s explore them 👇
12/n
🎯The central topic: how alternative assessment was done in lieu of actual exams
The assessment has 3 components–
1. Teacher Assessed Grades (TAGs)
2. Ranking
3. Algorithmic Standardization
Let’s explore them 👇
12/n
Act 2, Scene 1: Teacher Assessed Grades (TAGs)
⏲️Without real exams, historical grades were used
🤯But BD students take a diff exam prep path where past data isn’t representative of future performance
⚡️To meet unfair data demands, unfair & arbitrary assessments emerged
13/n
⏲️Without real exams, historical grades were used
🤯But BD students take a diff exam prep path where past data isn’t representative of future performance
⚡️To meet unfair data demands, unfair & arbitrary assessments emerged
13/n
Act 2, Scene 2: Ranking
🤨Teachers had to rank in a rather peculiar way– no ties were allowed
🤯No one knew the “why” behind this requirement
💯This way of ranking fundamentally changed how teachers assigned TAGs
🎯The rank order will play a crucial role in the imprint
14/n
🤨Teachers had to rank in a rather peculiar way– no ties were allowed
🤯No one knew the “why” behind this requirement
💯This way of ranking fundamentally changed how teachers assigned TAGs
🎯The rank order will play a crucial role in the imprint
14/n
Act 2, Scene 3: Algorithmic Standardization
Two main problems:
1. No details of how the algorithm worked were shared before grades were due
2. No one knew that the reason for the ranking without ties was b/c the algorithm couldn’t function without that type of input data
15/n
Two main problems:
1. No details of how the algorithm worked were shared before grades were due
2. No one knew that the reason for the ranking without ties was b/c the algorithm couldn’t function without that type of input data
15/n
🪧 Act 3 captures the events of Results Day (Aug 11) when protests broke out after the algorithmic grades came out
💢Unable to direct anger at the faceless exam boards, the teachers were caught in the cross-fire and became targets of abuse, causing psychological trauma
16/n
💢Unable to direct anger at the faceless exam boards, the teachers were caught in the cross-fire and became targets of abuse, causing psychological trauma
16/n
Act 4 covers grade revisions
▶️The boards revised the grades by 1) dropping the standardization & 2) assigning TAGs as the final grade
🤨The story appears to have a happy ending, but was the algorithmic effect truly undone?
🎁The algorithmic imprint can help answer this
17/n
▶️The boards revised the grades by 1) dropping the standardization & 2) assigning TAGs as the final grade
🤨The story appears to have a happy ending, but was the algorithmic effect truly undone?
🎁The algorithmic imprint can help answer this
17/n
Here’s how the imprint remained even after grade revision
Originally, we had 3 things
1.TAGs
2.Ranking
3.Algo Standardization
💡During revision, they removed algo standardization and assigned TAGs as the final grade
⚡️But remember how ranking influenced TAGs?...
18/n
Originally, we had 3 things
1.TAGs
2.Ranking
3.Algo Standardization
💡During revision, they removed algo standardization and assigned TAGs as the final grade
⚡️But remember how ranking influenced TAGs?...
18/n
...Remember how ranking had to be non-tie-based so that the algorithm could run?
🤯The ranking was a REMNANT of the algorithmic requirements
🔥It was part of the imprint & strongly impacted the end-result (revised grades=TAGs) even when the standardization was removed
19/n
🤯The ranking was a REMNANT of the algorithmic requirements
🔥It was part of the imprint & strongly impacted the end-result (revised grades=TAGs) even when the standardization was removed
19/n
🌟Like the remnants in a palimpsest, the imprint of an algorithm persists in its afterlife
🔥 Algorithmic harms thus can extend far beyond the algorithm’s lifetime
🎯This is one of my favorite quotes capturing the essence of the imprint:
20/n
🔥 Algorithmic harms thus can extend far beyond the algorithm’s lifetime
🎯This is one of my favorite quotes capturing the essence of the imprint:
20/n
![“It’s as if the ghost [spirit] of the algorithm lived on](https://pbs.twimg.com/media/FVTpn1BWUAEmWYQ.jpg)
Finally, reflections
💡 Algorithms have always been imprint-laden
💡 Imprints do not change their existing nature
💡 The concept expands how we view algorithmic impact.
💡 It provides traction to address ethical issues in the algorithmic afterlife– an unexplored area
21/n
💡 Algorithms have always been imprint-laden
💡 Imprints do not change their existing nature
💡 The concept expands how we view algorithmic impact.
💡 It provides traction to address ethical issues in the algorithmic afterlife– an unexplored area
21/n
💯I had the distinct pleasure of writing this paper with an amazing team – Ranjit Singh & @undersequoias from @data_society, @mark_riedl from @mlatgt @GeorgiaTech
🎁 Beyond grateful to this team for going on this crazy journey with me & trusting me throughout the process
22/n
🎁 Beyond grateful to this team for going on this crazy journey with me & trusting me throughout the process
22/n
On a personal level, this project means a lot to me.
💌 As a Bangladeshi & a product of the GCE system, I wanted to make sure Bangladeshi voices were not erased from the Ofqual scandal's narrative
🙏 Immensely thankful to everyone who helped along the way.
23/n
n=23
💌 As a Bangladeshi & a product of the GCE system, I wanted to make sure Bangladeshi voices were not erased from the Ofqual scandal's narrative
🙏 Immensely thankful to everyone who helped along the way.
23/n
n=23
• • •
Missing some Tweet in this thread? You can try to
force a refresh
