Thread by @abbott567

I've recently been collecting accessibility audits to try and understand what makes a good one, what makes a bad one, and if there is any work we can do to try and standardise them a bit, at least across Government. Here is a thread of my findings.

The World Wide Web Consortium (W3C) outlines that an audit should include 8 things:
1: Executive Summary
2: Background
3: Scope
4: Reviewer(s)
5: Review process
6: Results and recommendations
7: References
8: Appendices
w3.org/WAI/test-evalu…

I got sent a whole bunch of audits, so a big thank you to everyone that got involved! A lot of them were conducted by the same organisations so looked at 10 audits by 7 organisations. The additional 3 are significant iterations to the format or process by 2 of the organisations.

Out of the 10 audits I studied, 7 of the websites belonged to organisations in the Public Sector and 3 belonged to organisations in the Private Sector.

Of the 10 audits I studied, 4 of the audits were conducted by organisations in the Public Sector. 6 were conducted by organisations in the Private Sector.

Of the 7 organisations I studied, 2 out of 4 Public Sector organisations chose to do their own audit work, and 0 of 3 Private Sector companies chose to do their own audit work.

Of the 10 audit reports I studied, they were supplied in 4 different formats. They are as follows:
6 are PDFs
2 are spreadsheets
1 is HTML
1 is a slide deck

I compared all 10 of the audits against the 8 criteria recommended by the W3C. I was a generous as possible, for example an organisation may not have explicitly stated the scope, but it could be determined as they provided a spreadsheet for each page they tested against.

Out of the 10 reports:
7 had an Executive Summary
7 had background info
10 had the scope of the review
7 had the reviewers name
but only 4 had the reviewers contact info
5 had the review process
10 had results
but only 6 had recommendations
1 had references
4 had appendices

No single audit contained all 8 of the W3C recommendations for what should be included. The only constant feature in every audit was the scope and the results, but as mentioned earlier I was generous with my marking.

The value of the results was also widely varied. Some reports just listed 'pass' or 'fail' for an entire page, and some provided a granular explanation as to the exact part of the page causing the fail plus a screenshot.

Ironically, a few of the accessibility audits were inaccessible. PDFs, slide decks and spreadsheets can be difficult to navigate with assistive technology. Some were better than others, but only 1 of 10 was formatted as HTML.

Accessibility audits are difficult to study as so few organisations produce them. This may be incorrect as it is just my experience, but of all the audits I was sent I only managed to find 7 organisations producing them.

There is no standard way of producing audit reports. This makes them difficult to comprehend and compare.

The majority of the audits I compared were for websites owned by Public Sector organisations. This makes sense as they're mandated to meet the Public Sector Bodies Accessibility Regulations 2018

The public sector organisations which provide their own audits tend to contain less of the W3C recommendations. This doesn't mean the audits are worse. It's just likely that the additional effort of including references etc provides little value to the org.

Some Private Sector reports contained recommendations, but they were of no value. The recommendations were basically just 'pay us to fix it for you'.

That concludes my thread. If anybody would like to discuss anything I've mentioned please let me know. I'd also be interested in any findings you might have that align or contradict anything I've outlined. Thanks =)