Thread by @LeaKissner

A few years back, Abhishek Parmar (at the time, the TL of Bigtable) and I started Zanzibar to handle access control for Google products. Like... basically all Google products.

The paper is public (and will be in ATC '19):
ai.google/research/pubs/…

Other bits not in the paper 🧵🧵

Zanzibar provides a uniform data model and configuration language for expressing a wide range of access control policies from hundreds of client services at Google, including Calendar, Cloud, Drive, Maps, Photos, and YouTube.

[Z] scales to trillions of access control lists and millions of authorization requests per second to support services used by billions of people It has maintained 95th%ile latency of less than 10 milliseconds and availability of greater than 99.999% over 3 years of production use

The paper describes the interface and technical design & implementation. What it doesn't describe (and I should go and write this down) is *why* the interface is designed how it is. The short version is the access control semantics are designed to do a few things:

1. The access control model is designed to be flexible in many ways, like allowing you to express. "this image should be viewable by Alice as well as whoever can view the document in which it is embedded". It's also carefully *not* flexible in certain ways to avoid user confusion

2. The access control semantics are designed to be strongly consistent. For those of you who don't work in distributed systems, "consistent" is a word with a special meaning: different parts of the same system give the same answer.

en.wikipedia.org/wiki/Consisten…

A lot of systems use a model called "eventual consistency" which is "um, all the different bits will catch up eventually but until then you might get the old answer". This semantic does *not* mix well with access control!

Eventual especially does not mix well with a world where access control data lives in one place and the actual data lives in multiple places. To avoid that, Zanzibar is strongly consistent not just with itself, but has a feature to coordinate with object data changes.

... This is really hard to describe in a Tweet thread. But please, read the paper and let me know what you think of the interface and access control semantics. They were my best thinking at the time and seem to work well, but access control is still not a solved problem.

... I just noticed the Wikipedia page on "consistency model" is labeled as "may be confusing". Given how much 🤯 I get every time I explain to people that distributed storage systems often aren't consistent, that seems about right.

There's also a story behind that project name. That is not the original project name. The original project name was force-removed by my SVP. Once my hands are free again, I can explain.