Accessibility testing is quickly becoming one of the most important parts of a software tester’s toolbox. As the federal government and more and more provinces begin passing legislation mandating web accessibility for its residents, everyone in the digital space is looking into how they can ensure that all users have equal access to everything on the internet, from websites to games, to apps.
Building accessibility into every step of your process, from ideation and design, through development and testing to production requires careful consideration. So if you are kicking off a new web project, doing some research on how to incorporate accessibility principles should be top of mind. We’ve got accessibility sessions that can help with that if needed, give us a shout! But what if you have already a website, application, or platform up and running?
Still brushing up on your accessibility fundamentals? Check out our accessibility cheat sheet to understand the basics.
You may look to any number of free accessibility checking tools that have popped up recently. Most work the same way by asking you to provide a website URL on their site and then an email address where they will send your accessibility report. The whole scan process takes no more than 30 seconds and usually provides you with an inventory of compliance for your webpages along with an overall score.
The majority of these reports will show that sites need improvement to become more accessibility compliant, which often means bringing in designers, developers, business analysts, and others to reassess and rethink elements of, in this case, a website. Certainly strengthens the case for considering accessibility earlier in the process!
While the free accessibility checking tools are a good first start (and who doesn’t love something free?) if you are serious about accessibility and want to have full coverage on Web Content Accessibility Guidelines (WCAG) in specific standard (2.0/2.1) with (61/78) success criteria and compliance level (A/AA/AAA), then you may want to consider the benefits of bringing in an independent accessibility testing team to help. The free accessibility reports can definitely give you an idea where to start by identifying direct violations of WCAG success criteria, but an experienced accessibility testing team will ensure that full test coverage is planned and executed with the correct mix of automated scans and dynamic manual tests from accessibility experts.
While an accessibility testing team will have more upfront costs, the benefits will outweigh the risks in the long run, giving you the confidence of knowing all users, regardless of their ability, are able to experience your digital presence exactly as intended.
Onboarding a dedicated accessibility testing team can bring you the following benefits:
- Develop an organization-wide accessibility testing strategy
- Raise accessibility awareness from the top of the organization down through accessibility training sessions
- Develop a testing approach based on your application technology landscape and product usage
- Ensure you are relying on the correct mix of accessibility tools
- Report accessibility issues in a specific WCAG testing report and
- Help in defect triaging by reviewing false positives, a key downfall of the automated scans
There are many elements that need to be considered when doing accessibility testing. We’ve put together a helpful guide that gives you an idea of where the free tools work and when it may be time to bring in the accessibility testing experts.
| Accessibility Team | Free Accessibility | ||
|---|---|---|---|
| 1 | Clearly define the scope of testing | An accessibility team can help you clearly identify the scope of your project for this non-functional testing. | Free scans cannot help in finalizing the scope of your project. |
| 2 | Develop test strategy with sampling approach based on client requirement | An accessibility team can help you develop a test strategy to identify the correct test approach to meet WCAG standard requirements. | Free scans cannot build a test strategy for your organization. |
| 3 | Develop test plan based on your Project plan | An accessibility test can develop a test plan based on an approved test strategy so that the project teams have a clear idea of scope, timelines, tools, and test entry-exit criteria. | Free scans cannot build a test plan for your project requirement. |
| 4 | Clearly detects all major WCAG errors to the A level of WCAG | An accessibility team will run automated tests and then review results against manual tests. Having an actual person will also catch certain errors – e.g. an image might have alt text and pass the free scan, but if the alt text is wrong, the scan won’t catch it and your user will be given the wrong information. | Your accessibility scan can provide similar category of static issues in your application presentation layer in much faster rate but there can be high number of false positive issues. |
| 5 | Detects colour contrast issues | The accessibility team will rely on automated testing to detect colour contrast issues. | There are few WCAG criteria which are best suited to be picked up by free scan and machine code. The free scans are excellent at identifying these colour contrast issues. |
| 6 | Detects compatibility with assistive technologies like screen readers | Relying on a combination of free and paid tools, the team will check whether your digital experience is set up for those relying on a screen reader. | Most free scans don’t offer this. |
| 7 | Detect usability and design issues | Much of the work in identifying accessibility issues with usability and design issues (form errors, colour-driven information sharing, timers, text zooming) will only be detected by manual accessibility testing. | The free scans will be able to identify many of these issues, but not all. This can produce a high number of false positive. |
| 8 | Detect audio and visual related WCAG success criteria | An accessibility testing team will do a thorough review of all audio and visual elements to ensure they are accessible. Note: Many of the errors in this category are A level WCAG. | Most free scans don’t offer this. |
| 9 | Check platform accessibility (Window/Mac/Android /IOS) and cross browser compatibility | All relevant, in-scope browsers and platforms will be checked for accessibility. | The free scans are primarily geared toward checking only websites on the browser of your choice. This may not be the only channel where your users are coming from. In order to have full coverage, a proper scope of browsers and platform is needed to be planned in the test strategy. |
| 10 | Performs accessibility scans on documents | A thorough accessibility team will review any documents that are part of your digital experience to check for accessibility compliance. Some will even perform remediation services to update them. | While there are tools online that will do this, they are different from the free accessibility scans. For example, PDF accessibility testing and remediation can be achieved using a commercial version of PDF Acrobat pro. |
| 11 | Links all errors to relevant WCAG success criteria | The report from a dedicated accessibility team will link each of the identified issues with their corresponding WCAG success criteria so that you are able to understand where the errors are and how they can be addressed. | While some of the free scans will provide this information, many do not. Also please be aware that free scans do not provide full WCAG coverage and generally produce a report that is limited to a few of the WCAG success criteria, usually not more than 25-30% of overall WCAG scope. |
| 12 | Automating accessibility regression tests | An accessibility team can help develop automated accessibility tests in your existing automation framework or develop a new one from the scratch that will generate automated test results as you make new changes in your application codebase. | There are free and paid automated tools in the market, and your team should be able to apply them in an automation framework. |
| 13 | Accessibility Test Summary reports | An accessibility team can help you develop and share your testing status and summary report in a standard accessibility format which is unlike a mainstream testing report. An accessibility testing report should clearly showcase issues and findings linked to WCAG requirements and the project scope. | The free automated scan lacks this function to develop accessibly test summary reports. |
Both dedicated teams and free scans have their place in accessibility testing. Wondering about how your own digital presence stack up? Give us a shout.
Abhishek is a QA evangelist who is passionate about quality assurance and testing at all levels of the organization. He is currently the Director of Service Delivery, Ontario, and also leads Web Accessibility TCoE at PLATO. Abhishek is PMP and has played key roles throughout his career in positions like Service Center Manager, Delivery Manager, QA Portfolio Manager, and led Managed Services Testing Teams spread across the globe. Abhishek loves to train and coach teams in software testing and its principles.