Thread by @dvassallo

OK, I decided to build this and try to make a living off it:

An end-2-end encrypted serverless platform for web/mobile apps, offering user auth, db, & file store services. The end user gets privacy. The app developer gets spared from the liability & burden of handling user data.

https://twitter.com/dvassallo/status/1122968287355129856

Neither the app developer, nor me (the platform) will be able to see any user data. (Same as what @1Password does.) This also means automatic GDPR compliance for the app developer. I'll share a lot more technical and business details in a few days. I'll also open source it. ✌️

Here's the mailing list if you want to keep up to date. No more than 1 to 2 emails a month. updates.encrypted.dev/subscribe 🙏

Some more details:

The platform will offer 3 core services:

1️⃣ Account Management
2️⃣ Database
3️⃣ File Store

All these will be available through a JavaScript API for the browser. Initial iOS and Android support will be through @reactnative.

A user signup generates an asymmetric key pair in the browser. The public key gets sent to the platform along with the username & password. The private key gets stored in the browser’s local storage and downloaded to be put somewhere safe, or printed as a QR code.

User authentication happens through a process similar to passwordless ssh: The platform encrypts a random message with the user’s public key that only the holder of the private key can decrypt. The decrypted message gets sent to the platform, and a match authenticates the user.

The authenticated session gets kept alive through a persistent WebSocket connection between browser and platform. All further activity between browser and platform, or vice versa, will go through this authenticated WebSocket.

When a user gets created, a db partition automatically gets created for that user. When a user adds/updates/deletes db records, the statements get encrypted on the client with the user’s public key, and then the platform simply appends those encrypted statements to the partition.

The db partitions can be thought of as encrypted transaction logs. So when the user queries the db, the client has to get the full transaction log for its partition, and also decrypt it with the user’s private key. Then all queries are made client-side in the browser.

As a perf optimization, the user can have more than one db partition. For example, in a project management app, each project could be its own partition. That way, when querying one project, only the records for that project get downloaded and processed.

Another perf optimization is having the client materialize the results of certain queries, encrypt the results, and store them in the platform. Then when the query gets re-run, the most recent results get merged with any new transactions, rather than process all from scratch.

The transaction log can also get compacted (remove deleted records, collapse all updates) by the client and re-uploaded to the platform (encrypted, of course).

Local storage can also be used to cache materialized queries or compacted transaction logs. (Albeit capacity is small.)

The db will be a NoSQL schemaless JSON store. The JavaScript SDK will support some basic querying operations, such as filtering (select & where), aggregations (group by), and sorting. Anything more complex can always be done by the app.

A user can grant other users access to a db partition it owns. For example, in a chat app, a user could create a partition for a chat room, and grant read/write access for that partition to everyone in the organization. Then everyone in the org can post to the same chat room.

Real-time updates, such as chat messages or push notifications, can be thought of as a live db query that never ends. The client can subscribe to receive any new transactions added to a db partition. The platform would push them through the already-established WebSocket.

Finally, the file store is just blob storage encrypted with the same keys. The main use case is for storing user uploaded files. The interface will be that of a key/value store. Files can also be shared with other users in the same company/org/family, just like db partitions.

Next time I’ll describe how sharing would work in more detail. And if you want to keep up to date about the project’s progress you can also join the mailing list here: updates.encrypted.dev/subscribe. 🙏

Let's go over sharing:

First of all, user data is encrypted so that:

1⃣ the end user gets privacy from the app developer and platform.
2⃣ the app developer does not get burdened with the liabilities of handling user data.

So it's probably ok for share groups to use the same 🔑

Encryption isn't there to ensure privacy from other users in a group. Traditional user authorization (access control) will handle that.

And with that I have to make a correction. Earlier in this thread I said that user authentication would be done using just the private key. No.

That wouldn't work when multiple users share the same private key. So I think user auth can be done using a combination of the password and private key. This way different users sharing the same private key can still be uniquely authenticated.

Say there's a project management app using this platform. All users in the same company could share the same public/private key. If it's a big company, there could be a key pair per organization or per team. This does not invalidate reasons #1 & #2 for encryption.

The app could also support having multiple keys installed so that if Team A encrypts with key A, and Team B encrypts with key B, and they later decide to work together on a project owned by Team A, then Team A can give Team B their key and Team B would use both keys in their app.

There are some applications for which this obviously wouldn't work. For example, you won't be able to share access with anyone using just an email address. You'd have to exchange the key first. So, you can't build, say, Facebook on this. That's okay.

This works well enough for information systems where share groups are made of related ppl: project management, group chat, internal CMS/wikis, CRM, code reviews, vehicle tracking, invoicing, financial analysis, web analytics, health monitoring, code repo, family photos, ERP, etc.

One downside is that changing the pub/private keys would log out everyone in the share group & would require everyone to register the new key before logging back in. This is not great, especially in scenarios where key rotations can happen frequently, such as employee departures.

So maybe the solution is to denormalizing all db & file data for every user in the share group. i.e. each user would have a full copy of all the shared data. This seems inefficient but it might be the right trade off. I believe this is what @1Password does support.1password.com/secret-key-sec…

At the cost of more storage space, this would ensure that each user has all the accessible data encrypted with the user's own unique key (which would be a combination of the user's password & secret key). Now key & user revocations would become simple. Or am I missing something?

135 people subscribed to the mailing list from just this tweet thread. Much more than I expected. 🙏❤️

Next step is a scrappy working proof of concept on GitHub. That will hopefully give the project some attention from the Hacker News & Reddit crowds too. 🤞

The target for June is a working proof of concept. Here's the plan of record: public.3.basecamp.com/p/Y8xjrvcwfwDt…. It's a live document so it will evolve over time.

Thank you  for starting to educate people about how user data should be handled. techcrunch.com/2019/06/03/app…

Proof of concept coming along well.

Here's an end-to-end encrypted Todo app. DynamoDB is holding the transaction log of all db operations, and the record field is encrypted with a 🔑 that never leaves the user's browser.

This week will focus on perf. Will update with results.

Perf tested the todo prototype app this week with 1K 10K & 100K items. Initial loading latencies for the client-side DB are 0.07s, 0.9s, and 9s resp. Now working on some significant optimizations, but perf already decent. More details to follow soon, here and on the mailing list.