Profile picture
, 17 tweets, 5 min read Read on Twitter
Next talk at #pepr19 will be about "mandating the forbidden", and implementing privacy by design in production abuse systems, by Andy Schou from @Google.

I'm super exciting about this: privacy/security trade-offs in abuse systems are crucial and not discussed nearly enough.
@Google Anti-abuse system used to be a feature. It's becoming a mandate.

The speaker mentions the "Christchurch call":
@Google Reported users are users. Their data is user data, and legitimate users are often reported. You can't just make abuse a completely separate thing.
@Google General privacy needs still applied. Data should be purposed-limited, you need to limit data access correctly, deletions need to be propagated correctly.
@Google In abuse systems, there are specific privacy challenges. The first is to maintain a source of truth.

One example: keeping spam email content, to prevent it from being used in spam campaigns again.
@Google You need to minimize access correctly. Abuse systems are going to help respond to subpoenas and regulatory inquiries. Abuse analysts and raters need to be able to write rules and examine trends.
@Google Abuse systems contain exceptional processing paths, that you have to make sure are not used for any other purpose.
@Google You can address this with design patterns:
- mostly-transient processing and data minimization
- semantic annotations and granular access controls
- siloed data-processing exceptions.
@Google How do you keep canonical stores canonical?

Fighting abuse often requires purpose-limited processing, and keeping everything inside a silo gives you much more control over APIs.
@Google Normal deletion behaviors can obscure abusive patterns. You still want to delete things from your canonical repository; so it can't be the one you use for abuse systems (where you need longer retention for some things).
This sums up these two possible paths of deleted data with abuse systems.
And there's still data you need to keep, for example:
- golden datasets
- abuse verdicts
- trust scores
- appeals data
You also have to keep access minimal and auditable. How do you do it?

First, you add semantic annotations to your data, to know what each field contains. Things like "this int64 is a pseudonymous ID".
(Woa, this talk is making many more things public than I expected!)
Then, you build access controls around these semantics annotations.

This allows you to check e.g. that analysts only use the data that they need when running queries.
You want your abuse systems to ingest only the data you need and only when you need it.

The intake process can be event-driven, like with automated scanning or user reports.
And you want to keep it in the least sensitive form possible. If you don't need the user-generated content to build your abuse-fighting intelligence, then throw away the content, but keep the scores.

When designing exceptional processing, narrow the output as much as you can.
In summary: make guarantees (invariants?) predictable, with org-wide retention practices. Centralize your exceptional processing, and there, make decisions about retention only after a verdict is reached. Label your data, use it for granular access controls.
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Ted
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!