Thread by @michaelbolton

I'll be live-tweeting Fiona Charles (@FionaCCharles)'s talk at #EuroSTARConf. Here's the thread.

@FionaCCharles Fiona fell head over heels in love with software starting in 1978. She is becoming increasingly concerned with the way that we're letting software loose on the world. She's going to share concerns about that.

@FionaCCharles Let's start with Amazon and its face recognition. It's particularly bad with the faces of women, black people, and especially black women The technology is being sold to police forces around the world... with up to 98% false positives.

@FionaCCharles What's even more concerning is how it's being used to evaluate people, not just to label people. Psychologists will tell you that it is not possible to read people's emotion reliably by the face. Now this stuff like this (in a $20 billion market) is being used on kids in China.

@FionaCCharles Fiona was inspired to do this talk after reading an article on AI that would label people as gay or straight. What could POSSIBLY go wrong with that, in places where that's a potentially lethal evaluation? Recent events (e.g Apple Card) show the biases.

@FionaCCharles It's not about the data; it's also about the decision criteria. Here's another bias: service droids, robots, being used to substitute for wait staff at restaurants in Asia. Significantly, they (like Siri, Cortana) are all female. Who else would do the drudge work?

@FionaCCharles You can choose to change these things, but the default is female.

We are moving to a cashless society, where we're paying with phones and cards. Lots of people (old, poor,...) will be left out of the cashless society.

@FionaCCharles AirBnB, wonderful as it may be, is causing big problems in cities around the world by displacing rental stock all over the Western world. That's not the technology's or company's fault per se, but the effect is that it's harder for people to live where they work.

@FionaCCharles Next: Cambridge Analytica, and the big, well-financed attempt to influence elections and politics in the UK and America. Brexit was the proof of concept for this. Personal data got picked up and analyzed for personal psychological data for virtually every voter.

@FionaCCharles CA used this data to micro-target ads for their (exclusively) right-wing clients. This didn't happen in the open; it wasn't for public debate. We don't even know if this data theft and manipulation of the electorate works. But presumably it does; it's being continued worldwide.

@FionaCCharles Data is now being used from smart toilets (???!!), from robot vacuums, from health monitoring gizmos. Very personal data (e.g. on women's periods) is being gathered. History suggests that this will not go well. From monitoring exercise data, military secret bases were located.

@FionaCCharles We are all carrying devices that allow (e.g.) abusive men to stalk women, or to harass them through home control apps for lights and heat. Plus the security on these things is appalling. It's scary when your house seems to be attacking. The iPhone as a very detailed, bury list..

@FionaCCharles ...of where you can be located.

Employers now have stuff that monitors where you are, whether you're walking or talking, active. The product is called, ironically, "Humanyze". (Fiona shows a video from the Economist.) Companies are now microchipping their employees.

@FionaCCharles But first you have to get the job... and there's face-scanning data that (dis)qualifies you for the job — even if you want a job as a babysitter. The software can supposedly tell if you have a "bad attitude".

@FionaCCharles Announcements are blithely made about disappearing work, without asking how people will survive and who will pay.

We're addicted to convenience. The number of delivery vehicles on the road has increased dramatically, adding to pollution and accidents.

@FionaCCharles Next, "Anymal" (a robot of the kind discussed by @atscmc yesterday, with the human controller just out of shot). Dog-like robots touted as "a smart way to overcome the last mile of delivery". These robots are creepy. They're always on empty streets in the videos.

@FionaCCharles @atscmc People will be very annoyed by these bots. A recent story from Twitter: a disabled woman could not get out of the intersection because the robot was occupying the curb cut.

Uber and Lyft are increasing, rather than reducing, the amount of traffic by a staggering amount.

@FionaCCharles @atscmc The taxis and other traffic is still there; we've now added the ridesharing vehicles, cruising and waiting for a call.

When we buy stuff online and return it, the returned stuff often goes to the last one. The convenience for us has its consequences for everyone.

@FionaCCharles @atscmc Hackers have been shown able to evade the dismal security to get control of the car via the entertainment system.

On home routers from Bell Canada, routers come with default username/passwords "admin/admin", but don't tell people.

@FionaCCharles @atscmc Story of Qantas flight from 2008: the plane dropped several thousand feet. The initial explanation was clear-air turbulence. Turns out if was a flight control computer error; turning the nose down over 10 degrees in a second. The Airbus A330 here was a fly-by-wire plane.

@FionaCCharles @atscmc These planes can block the pilot's attempts to regain control of the aircraft. These systems have helped flying to become safer... when it works. When it doesn't work, faults can happen and are hidden often from the pilot's view.

@FionaCCharles @atscmc What level of control should we relinquish? The 30,000-hour pilot of this plane had been a military pilot, but had never felt fear or lack of control.

What can WE do about all this?

@FionaCCharles @atscmc Answers: we MUST keep asking "What could possibly go wrong?" We need to question our assumptions; ask "should we build this thing at all?" We're building software because we can, because it's fun, because we can make money. Is all that good enough?

Look for possible biases.

@FionaCCharles @atscmc Acknowledge the limitations. Data is just data. Stop trying to replace skilled operators with technology. Keep the operators of the technology engaged. When humans are less involved, their involvement when needed becomes more critical. Build security in.

@FionaCCharles @atscmc Ground tech in human society. Help people who are not involved in development to understand the technology and its limitations and dangers. Study and get involved and contribute to ethics in tech. Remember the small things that make up the fabric of society.

@FionaCCharles @atscmc By automating things and putting screens between people, we are may be undermining society.

The military assures us that new weapons "adhere to the AI Murder Policy". AI is now being applied to nuclear weapons management.

What could possibly go wrong?

-fin-

@FionaCCharles @atscmc Q. What can we, as testers do?

A. One of the most important things we can do is to ask what could possibly go wrong at every step. That isn't a primary consideration; we're seeing wonders and benefits and money. We can ask about problems and develop scenarios about Bad Things.

@FionaCCharles @threadreaderapp Unroll, please!