Thread by @Contextify1

How I'd improve AWS' documentation, a (probably very long thread) 🧵

https://twitter.com/heitor_lessa/status/1329830499611766785

Let me start with my perception of _why_ people read technical docs:

1. To evaluate if they need something
2. To set up something (presumably only once)
3. To troubleshoot setup or implementation
4. [a distant fourth] To learn more about underlying concepts and principles

go to docs.aws.amazon.com

Does it meet any of those needs?

Because they have _so many_ offerings, the only effective way to navigate is through search, which is relegated to a slim header

Nobody will read the full docs of 10 databases to pick one

If I were designing the docs home page, it would just be:

1) giant search bar in the middle
2) big links to
2a) Comparisons
2b) Guides
2c) Examples
2d) Troubleshooting
2e) Those Verbose Machine Generated Docs They Currently Force-Feed Us

Let's break out each one.

Comparisons

a) Write a short explainer - for each axis on which decisions are made - comparing the most relevant services (e.g. which storage service to use for high vol writes + low cost)

b) Explain tradeoffs (e.g. cheaper but hard to setup?)

c) give clear, lay explanations of cost. I don't want to have to back out what a Dynamo WCU actually nets to. Have two example usage volumes and lay out each service's cost side by side.

d) Really, really try to write as if the reader doesn't know the space. They probably don't!

Guides:

This is the one thing AWS is closest on, but their discoverability is so poor.

a) title everything How To Solve This {Business|Technical Problem} With These Services - [A,B,C]

b) state upfront the design constraints. I want to know why these services were chosen.

c) state upfront reasons why you wouldn't use this architecture. If I need a $100/mo instance to run it, it is inappropriate for most non-Big-Co-In-Prod usecases

d) try to avoid shiny-object-ing in new services (unless relevant). AWS cares a lot about Polly & Sagemaker. I don't

e) go search the internet for good Medium/dev.to/personal blog guides. PAY THOSE WRITERS TO DO MORE. They are so consistently so much better.

and finally f) Replace every single IAM "*" and VPC "0.0.0.0/0" with an equivalent as downscoped as possible

Examples

For examples, I just want to see one service do one thing (e.g. trigger a Lambda with a change to an S3 bucket)

Look at @goserverless 's docs. One page. Straight to the point. Useful. serverless.com/framework/docs…

Now look at the AWS SERP. Which one do you even click?

I'll be clear: I have literally no idea how to find that example in AWS' docs _without using Google_. And I've used Lambda for years.

For serverless.com, it's Docs -> provider CLI -> AWS -> AWS Events -> S3

Here, again, the breadth of AWS hamstrings its Docs' UX

Let's dive in on what examples should look like

here comes a 🚨 CORE COMPLAINT 🚨

AWS supports 7 infra-as-code modalities and let's say 10 programming languages in the docs. What are the odds I find the 4 permutations I'm familiar with out of 70 total?

https://twitter.com/Contextify1/status/1329210260037398528

(counting the GUI as -as-code, lol)

There is one possible :chef-kiss: silver bullet here: let me set my infra- and programming-languages in the settings and default to that on any page that has it

the corollary to this is "you need to translate the examples into significantly more permutations"

Y'all have the usage data. Make sure the top 5-10 are in 80% of docs. Probably ok to skip Haskell + SAM, etc

This one feature would ~2x the usability of the docs

diatribe over, back to examples:

a) concise. avoid scope creep.
b) make note of dependencies, incl IAM
c) absolute minimum cross-selling.

d) use traffic to example docs to discover what use cases should get first party support in 1) boto and 2) accessory libs like datawrangler

To use the earlier SERP:

terrible (title is wrong, not relevant):
aws.amazon.com/premiumsupport…

bad (too general for this usecase):
docs.aws.amazon.com/AmazonS3/lates…

ok (provides event body and links to better docs):
docs.aws.amazon.com/lambda/latest/…

better (but kinda verbose):
docs.aws.amazon.com/lambda/latest/…

Which brings us to another 🚨 CORE COMPLAINT 🚨

As I said earlier, Google search is start of the vast majority of UX pathways (IME)

The titles and SERP placements of Docs are frequently not accurate or relevant. Do a full audit of bounce rate and time on site. Change the worst.

This, as well, would probably 2x the usability of the Docs.

I generally search with -site:aws.amazon.com because it ranks so highly and clogs the SERP with irrelevant/dense stuff

if I do use it, I open the first half dozen links in tabs b/c I know I'll close most.

final note on examples

please please please implement Toggle-Expand widely (pic 1)

If instead of this full screen of brackets (pic 2) you had:
A) hyperlink on inputFile.txt -> downloads that file
B) toggle expand

I could toggle-close after reading, ⬆️⬆️ rest-of-doc-readability

(it also significantly improves the UX of adding how-to-handle-this-edge-case sections, as you can put them in a default-closed toggle)

Most people won't e.g. deploy a 150mb zip to Lambda Layers, but the ones who do really need to see the It's-Probably-Bundling-Too-Much section

Troubleshooting

🚨 CORE COMPLAINT 🚨

Some error messages are a nightmare to search for

Truly, a Lovecraftian horror of Github issues with no response, Stack Overflow threads from 2013, an outdated blog post, and message boards.

I went through 163 tabs, 3 Slacks, and tagging a bunch of AWS people to still-not-be-able-to-solve this ridiculous Lake Formation IAM infection problem.

(more here:

https://twitter.com/Contextify1/status/1326330218710175745

)

To have the literal message have exactly 1 (third-party) result is unacceptable

So please: get a PM, make a team, & crawl the AWS production codebase for error codes and messages.

Put them in a database. Make that database queryable. Make that database search-index-able.

Note which ones:
1) have no dedicated AWS page
2) not mentioned in any other AWS pages

and 3) buy or scrape search data on
a) search volume
b) # of results

I'd bet my lifetime earning potential there's >100,000k/mo searches for errors with 0 AWS pages mentioning it and effectively 0 useful third-party results

100k, not 100,000k 😅

those searchers are the _most frustrated_ customers.

Any small improvements will be more valued by them than any other customers.

This would probably ~3x the usability of the docs.

In a perfect world, there's some Guides that focus more on broad troubleshooting.

The original Q from @rakyll was "what's the biggest friction with identifying and resolving latency issues?"

A: I have literally no idea where to start.

The SERP: VPC, VPC, VPC, ELB, Connect

that's OK if I have those problems, but they aren't relevant to my query

Why aren't there Guides for:
A) Why Is My API Slow
B) Why Do I Have Timeouts
C) When to use CloudWatch / XRay / a flamegraph
D) Here Are Benchmark Latencies

Honestly fine if they mostly link to other docs

Humans inherently don't know how to learn unknown unknowns.

How would you type into Google "P95 latency Cognito Custom Authorizer for 512 MB Lambda on HTTP API -"cold starts" "

if you barely knew 2 of the 7 concepts involved

small aside: even a query with a fraction of the above's complexity has very little coverage (44 results!)

in this case, AWS has ceded their authority to the omniscient @alexbdebrie. The Google SERP placement acknowledges that. (it is a very good guide tho)

Finally, Machine Gen'd Docs

For this, I'll reference @GoogleAds

Want to query your data? Bummer, you need a report

How do you pick the right report? Well, click into the ~50 reports, & they each have 30-50 fields. Just keep clicking until you find most of the fields you need

That's their old API. Their new API does it better:

A central list of every field, with a link to a dedicated page, and ✨ one human written sentence explaining what it's actually for ✨

& the search has gotten much better, which: why was it ever not good if it was by Google

back to AWS

You find yourself in (order of increasing frustration):
1) awscli docs
2) lang-specific lib like boto docs
3) AWS OSS Github repo
(god-forbid) a service's Developer Guide

Odds are, the main docs failed the user. Now they must CMD+F through (for ex) 835 pages!!

Best case it is a principal engineer with no urgency exploring a very niche feature or use case

Worst case some unknown IAM need took down prod and a very stressed junior engineer is firing off search queries and Slack messages

Which UX is worth prioritizing? 🧐

You can tell which teams matter to AWS by how well the 2nd UX is covered.

Look at aws s3 cp:
* it has awscli help
* it has options with argtypes with links for further reading
* it has examples with inputs and outputs!

Now let's look at my nemesis, #LakeFormation

- enumerates arg fields with this gawky indentation
- highly nested JSON format has no examples
- Has several inaccurate or nondescript error messages (eg "Permissions modification is invalid") that aren't mentioned anywhere

(you can start to see how the problems above combine to make larger problems)

I understand it's a lot to maintain
* awscli help
* docs.aws.amazon.com
* boto docs
* cli docs

but if you can't justify writing good docs, maybe don't ship the service?

OK, that's a lot of nit-pick-ery. Let's summarize.

1) if you're small, prioritize why-use-my-service and how-to-get-started in your docs. Do the rest over email.

2) implement search. Use it regularly. Is the UX better than Google? If no, why not?

3) pages should consider why users are reaching them. The terminology, example depth, even length of the page should reflect the user profile and circumstance

4) keep your docs updated. Highlight when each page was last updated. Deprecate pages occasionally.

5) Check the #'s for each post, benchmarked to similar ones:
a) page views (low -> nobody cares about this feature)
b) bounce rate (high -> irrelevant title/SERP or incomplete)
d) scroll depth
c) time on page
e) % exit (high -> good! their problem is (prob) solved)
f) SERP spot

and look at the metrics for that search you implemented in part 2

what queries come up most?
were they what you expected?

& do the same analysis on the queries as you did on pages

(guide for GA: moz.com/blog/5-actiona…)

and 7), you know, talk to your customers

talk to your customers' developers, if they're not normally the point of contact

give them a [e-mail] email that they can over-the-fence feedback into

Some things won't show up in [Google/internal] search queries b/c users don't associate it w/ you

I didn't know AWS had a simple DB (it's literally called SimpleDB) until 2+ years in

I didn't look because I don't associate them with simplicity

would have been handy then 😶

& 8) be out in the community, gathering unprompted feedback

for AWS, you see @quinnypig (rightfully) ribbing NAT Gateway & @mipsytipsy explaining how to move from poking-your-CloudWatch-logs to actual observability & @dvassallo making a fortune just by "explaining things well"

8a) I love the #AWSWishlist & think every company should have something similar, but it often requires getting an employee to publicly buy into it for it to ship

...makes sense at that scale

-> startups can get the best of both worlds w/ a lil keyword monitoring & transparency

[ been busy the past 24h moving, what a lovely surprise this has been to come back to ✨ ]

Appreciate y'alls interest in my Docs rant 😇

If you feel I missed / misrepresented anything, feel free to DM me

I will synthesize this into a proper blog post (sans Google Ads tangent 🙃) in the next month or two

For folks who want to learn more about API Docs, I highly recommend @swyx 's guide: swyx.io/documentation-…

My main takeaway:

I spent 44 tweets accosting AWS, and 8 AWS engineers showed up to patiently say

so from me to y'all ->

Postscript:

I'm currently writing API Docs for my own "stealth" startup.

When it hits public beta, I hope y'all roast it just as thoroughly 🥰