Measurable test coverage: Silver requirement? #38
Comments
It's only possible to claim conformance at the document/page level in WCAG, though a claim may encompass multiple pages. The practice of using a representative sample of pages to claim website level conformance is recognition of the fact that document/page level conformance for an entire website is impractical to the point of being impossible. The trouble with representative samples, as with any form of partial coverage, is what happens when the really important stuff is in the part of the site/page that isn't covered? We all know the expectation is that issues will be extrapolated from the test sample and resolved everywhere, but the reality is that this just doesn't happen. |
|
Perhaps a conformance claim also needs a standardised way to express what was tested. For example, could a website claim 100% coverage via automated tests and additional manual testing of 2 samples of each type of page, with conformance at 95% of WCAG 3.0 requirements. A small site might manage 100% coverage, but very large websites would really struggle to do more than representative testing. |
|
I've written the following document which describes how the conformance model works in Japan. We are using a representative sample of pages to claim website level conformance. JIS X 8341-3:2016 is the Japanese national standard which is originally ISO/IEC 40500:2012(=WCAG 2.0). FYI: Conformance model of JIS X 8341-3:2016 |
rachaelbradley
removed
the
section: conformance
rachaelbradley
added
the
migration: core
A challange in WCAG 1 and 2 is that it's impossible to say anything about conformance unless a web page is fully tested. Because you don't always have a complete picture, it is difficult to say anything about the accessibility. It'd be very helpful if in addition to allowing pages to conform if not 100% of the content passes the requirements, to also allow conformance claims of pages that aren't 100% tested.
This is going to be relevant when testing websites too. How much of the site should be tested before someone can claim conformance? Currently I think this is only ever done by counting numbers of pages. But I don't think that has to be the only way to do it.
This could in theory work if applicability of a requirement could be made programatically determinable. For instance, if a requirement is applicable to all img elements, and a page has 100 of them, with 90 img elements tested, you have 90% test coverage for that requirements. WCAG 2 doesn't have this concent of applicability, making a measurement like this impossible.
If we allow incidental issues in conformance of Silver, it makes sense to also allow test coverage to be lower than 100%. By writing Silver in such a way that it can be used to measure test coverage, it becomes possible to maintain conformance over time. Rather than have the conformance claim get outdated the moment a page changes, conformance claim can stay relevant as long as the test target didn't change too much. Even better, you can programatically determine the moment the conformance claim becomes outdated, as that will be the moment the test coverage drops below a certain level.
The text was updated successfully, but these errors were encountered: