Project 2 - Accessibility Assessments Analysis

Introduction

In order to evaluate systems or tools, we often use benchmarks . These are often a collection of assessments on which new and existing tools compete to perform well. For this assignment, we will be looking at the results of multiple tools (or “checkers”) that were run on a benchmark created by the United Kingdom’s Government Digital Services to test accessibility checkers. An accessible page is one that is usable by people with disabilities, many of whom will be accessing the web with assistive technologies such as screen readers or dictation software . The benchmark includes series of assessments, each with a single introduced error. These assessments are grouped into categories with other related assessments.

These assessments have been run on 13 different accessibility checkers and the full results have been made publicly available. For this assignment, we will use the a11yCheckersResults.txt file that contains the results of the assessments for only 4 of the 13 checkers.

a11yCheckersResults.txt format

The file is organized such that each line contains the results for one assessment organized as follows:

[category] [google result] [wave result] [sortsite result] [aslint result] [assessment description]

The first 5 parts of a line (the [1] category and [2-5] result of the assessment for each checker) are all a single “word”. The last part of the line is the assessment description, and it is of variable length.

The result of the assessment for each checker is a single word that indicates the result of the assessment. The 6 possible results are:

1. error
   • the issue was properly found
2. error_paid
   • the issue was properly found in the paid version
3. warning
   • the tool only gave a warning
4. manual
   • the tool required the user to check
5. identified
   • The tool noticed the issue, but didn’t give a warning
6. notfound
   • The tool failed to identify the issue

For instance, the line for the test that is used to see if a person can visually determine which element has keyboard focus looks like this:

Keyboard notfound notfound error error Keyboard focus is not indicated visually

This line indicates that the assessment is in the Keyboard category of assessments, the error was not detected by the Google or WAVE tools, but was detected by SortSite and ASLint.

In this PA, you will write code to read in the assessment results list and then use them to analyze different information about the assessments.

Part 1 - AccessibilityAssessment

You are to design and implement a class named AccessibilityAssessment, which stores the category and description of the assessment and the results of the assessment for the four checkers. The constructor for the class should take all 6 pieces of information as parameters and store those values in fields. It should have accessor methods for each value, e.g., getCategory, getDescription, getGoogleResult, etc. It should also have a toString method which presents a readable format of the results of the assessment.

classDiagram
class AccessibilityAssessment
AccessibilityAssessment: -String category
AccessibilityAssessment: -String googleResult
AccessibilityAssessment: -String waveResult
AccessibilityAssessment: -String sortsiteResult
AccessibilityAssessment: -String aslintResult
AccessibilityAssessment: -String description
AccessibilityAssessment: +AccessibilityAssessment(category String, googleResult String, waveResult String, sortsiteResult String, aslintResult String, description String)
AccessibilityAssessment: +getCategory() String
AccessibilityAssessment: +getDescription() String
AccessibilityAssessment: +getGoogleResult() String
AccessibilityAssessment: +getWaveResult() String
AccessibilityAssessment: +getSortsiteResult() String
AccessibilityAssessment: +getAslintResult() String
AccessibilityAssessment: +toString() String
AccessibilityAssessment: +equals(other Object) boolean
AccessibilityAssessment: +passed(checkerPartialName String) boolean

Constructor

The constructor is not required to validate its parameters. It should simply store the parameters in the corresponding fields.

toString(): String

This method should return a (1-line) String with the following information:

1. The category of the assessment followed immediately by - (a space, a dash, and a space).
2. The results of the assessment for each checker (in the order presented in the UML diagram and the constructor parameters) formatted as follows:
   1. The name of the checker (i.e. Google, WAVE, SortSite, or ASLint) followed by : (a colon and a space)
   2. The result of the assessment for the checker (from the previously enumerated list of valid result String values), followed by (a space).
3. The description of the assessment, preceded by - (a dash and a space).

equals(other: Object): boolean

This method should return true if the other object is an AccessibilityAssessment object with the same category, description, and checker results (for all 4 checkers) as this object. It should return false otherwise.

passed(checkerPartialName: String): boolean

This method should return true when the checker whose (case-insensitive) name contains checkerPartialName had a result of error or error_paid. It should return false otherwise. For example let’s say this AccessibilityAssessment object is representing the 2nd line of the provided a11yCheckersResults.txt file, which reads:

Content notfound notfound error_paid notfound Plain language is not used

In this case, we should have created an AccessibilityAssessment object with the following values:

• category: Content
• googleResult: notfound
• waveResult: notfound
• sortsiteResult: error_paid
• aslintResult: notfound
• description: Plain language is not used

So if we call passed("google") on this object, it should return false because the Google checker did not find an error. If we call passed("sortsite") on this object, it should return true because the SortSite checker found an error (in the paid version). If we call passed("wave") on this object, it should return false because the WAVE checker did not find an error. If we call passed("lint") on this object, it should return false because the ASLint checker did not find an error.

Part 2 - AccessibilityResults

The file a11yCheckersResults.txt as been provided for you, which contains the information about assessments used by the UK Government Digital Services to evaluate accessibility checkers. Each line in the file contains the information about a single assessment organized as described previously. You are to design and implement a class named AccessibilityResults that reads in the assessment information from a file of this format, stores them in an ArrayList of AccessibilityAssessment objects, and provides methods for accessing that information. When you are reading in the information, make sure that you are storing the values in the most appropriate types. Your implementation of AccessibilityResults should conform to the following UML diagram as well as the detailed specifications that follow. Note: you may assumes that none of the parameters will be null.

classDiagram
class AccessibilityResults
AccessibilityResults: -ArrayList~AccessibilityAssessment~ assessments
AccessibilityResults: +AccessibilityResults(filename String)
AccessibilityResults: +numTests() int
AccessibilityResults: +saveResults(filename String, format String, results ArrayList~AccessibilityAssessment~)
AccessibilityResults: +showTestResults(details String) ArrayList~AccessibilityAssessment~
AccessibilityResults: +showByCategory(category String) ArrayList~AccessibilityAssessment~
AccessibilityResults: +showAllFailed() ArrayList~AccessibilityAssessment~
AccessibilityResults: +passing(name String, category String) ArrayList~AccessibilityAssessment~
AccessibilityResults: +numPass(name String, category String) int
AccessibilityResults: +getAll() ArrayList~AccessibilityAssessment~

In particular, this class should have the following methods:

constructor

This method should take the filename as a parameter and should read in the file of accessibility assessments. The constructor should parse each line in the file to create AccessibilityAssessment instances and store them in the assessments ArrayList. The constructor should use try-catch exception handling to print an error message if an invalid filename is found. The error message should be formatted as follows: File not found: filename (where filename is the name of the file that was passed in as a parameter).

numTests

This method takes no parameters and returns the number of assessments that are stored in the ArrayList.

getAll

This method takes no parameters and (as already documented int he UML diagram above) returns an ArrayList. This ArrayList should contain all of the AccessibilityAssessment objects that were read in from the given file by the AccessibilityResults constructor.

saveResults

This method expects 3 parameters:

1. filename: The path to the file to which the results should be written.
2. format: The format string for the final line of the output.
3. results: the ArrayList of AccessibilityAssessment objects to be written.

This method should write each of the AccessibilityAssessment objects in the results ArrayList to its own line of the file specified by filename. These lines should be formatted according to the AccessibilityAssessment’s toString. The final 2 lines of the file should be:

1. an empty line
2. a line formatted according to the format parameter. This line will be passed a integer to support messages like, Total tests matching: 10 or Total tests failed: 51.

Use try-catch exception handling to print an error message if an invalid filename is found. The error message should be formatted as follows: File not found: filename (where filename is the name of the file that was passed in as a parameter).

showTestResults

This method takes assessment details (or a portion of the details) as a parameter, and both returns an ArrayList of AccessibilityAssessments that match (or contain) that detail (should be case insensitive) in either the category or description and writes those matches to a file (named as showTestResults-<details>.txt) followed by a summarizing message. The summarizing message should read as follows. For example, if the information from a11yCheckersResults.txt was read in and stored in a AccessibilityResults object named a11y, then a11y.showTestResults("Colour") should write the following to the file named showTestResults-Colour.txt:

Colour/Contrast - Google: notfound WAVE: notfound SortSite: notfound ASLint: manual - Colour alone is used to convey content 
Colour/Contrast - Google: error WAVE: error SortSite: error_paid ASLint: notfound - Small text does not have a contrast ratio of at least 4.5:1 so does not meet AA 
Colour/Contrast - Google: error WAVE: error SortSite: error_paid ASLint: notfound - Large text does not have a contrast ratio of at least 3:1 so does not meet AA 
Colour/Contrast - Google: error WAVE: error SortSite: error_paid ASLint: notfound - Small text does not have a contrast ratio of at least 7:1 so does not meet AAA 
Colour/Contrast - Google: error WAVE: error SortSite: error_paid ASLint: notfound - Large text does not have a contrast ratio of at least 4.5:1 so does not meet AAA 
Colour/Contrast - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Focus not visible 
Links - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Identifying links by colour alone
Forms - Google: notfound WAVE: notfound SortSite: notfound ASLint: manual - Errors identified by colour only
Forms - Google: warning WAVE: error SortSite: error ASLint: error - Errors identified with a poor colour contrast
HTML - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Inline style adds colour

Total tests matching: 10

showByCategory

This method takes a category (or a portion of the category) as a parameter, and returns an ArrayList of all of the AccessibilityAssessments that match (or contain) that category (should be case insensitive). It writes these matches to the file named showByCategory-<category>.txt. As before, the number of matching assessments should be displayed at the end. For example, if the information from a11yCheckersResults.txt was read in and stored in a AccessibilityResults object named a11y, then a11y.showByCategory("navigation") and a11y.showByCategory("nav") should write files containing the following (in files named showByCategory-navigation.txt and showByCategory-nav.txt, respectively ):

Navigation - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Inadequately-sized clickable targets found

Total tests matching: 1

Keyboard - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Dropdown navigation - only the top level items receive focus

showAllFailed

This method takes no parameters, and returns an ArrayList containing the AccessibilityAssessments that all checkers failed (i.e. an assessment is only shown if Google, WAVE, ASLint, and SortSite all failed it, and if even one of the checkers passed the assessment, it is not shown). The method will alsop write these AccessibilityAssessments to a file named showAllFailed.txt followed by a summarizing statement in the format specified below.

Content - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Content identified by location
...
HTML - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - Inline style adds colour
HTML - Google: notfound WAVE: notfound SortSite: notfound ASLint: notfound - PRE element without CODE element inside it

Total tests failed: 51

passing

This method takes two parameters, the checker name and category, and returns an ArrayList of assessments that had a result of either error or error_paid. The method will also write these AccessibilityAssessments to a file named passing-<checker>-<category>.txt followed by a summarizing statement in the format specified below. (Note this method should work with a partial checker name or partial category name and should be case insensitive). For example, if the information from a11yCheckersResults.txt was read in and stored in a AccessibilityResults object named a11y, then a11y.passing("Goog","") should write something like the following excerpt to passing-Goog-.txt:

...
Colour/Contrast - Google: error WAVE: error SortSite: error_paid ASLint: notfound - Small text does not have a contrast ratio of at least 4.5:1 so does not meet AA
...

Total tests matching: 23

numPass

This method takes two parameters, the checker name and category, and returns the number assessments that had a result of either error or error_paid. (Note this method should work with a partial checker name or partial category name and should be case insensitive). For example, if the information from a11yCheckersResults.txt was read in and stored in a AccessibilityResults object named a11y, then a11y.numPass("Goog","") should return 23 and a11y.numPass("lint","htm") should return 2.

Avoid redundancy and be conservative in your use of fields - if a data value does not need to persist over the life of the object, declare it to be local to the method that needs it.

Recommended Process

1. Create a new project in Eclipse called PA2.
2. Write and test AccessibilityAssessment.
3. Write and test AccessibilityResults.

AccessibilityAssessment

1. Define the instance variables
2. Write the constructor (if you wish, Eclipse can be helpful in generating a first draft of the constructor).
3. Write the accessors (“getters”). (Again, if you wish, Eclipse can be helpful in generating these methods.)
4. Test the constructors and accessors.
   • for this you may wish to start with AccessibilityAssessmentStudentTest.java
5. Write the equals method. (Again, if you wish, Eclipse can be helpful in generating this a first draft of this method.)
6. Test the equals method.
7. Write a test for what the toString method should do. (you should be failing this test right now)
8. Write the toString method. If you wrote your test well, you should be passing the test when you’re done with this method.

AccessibilityResults

1. Define the instance variable
2. Write the constructor
   1. Probably begin by having your constructor manually create a few AccessibilityAssessment objects and add them to the ArrayList.
   2. Once you have that working, you can start working on reading the AccessibilityAssessments from a file. Perhaps as you move into this step, you should create a smaller version of the data file, (maybe just make a copy of a11yCheckerResults.txt and delete all but the first 6 or so lines) until you’re happy with how it’s working, and then move on to the full file.
      • In reading the file, observe that while it may be helpful to know that (e.g.) your Scanner hasNextLine(), it might be helpful to extract less than a full line at a time , at least at first.
3. Write saveResults
   • note that this method is is early in the instructions, especially that it’s before other methods that should… write… results… to a file.

Additional Resources

JDK Scanner class

The Scanner class in the Java Development Kit (JDK) is useful for reading in data. It can help parse data from the command line as well as from a file. Take a look at the Scanner javadoc and also at the ArrayList lab (particularly in the Editor class) and the File I/O lab for examples of how to use the Scanner class.

JDK String methods

You may find several of the Java Development Kit’s String methods useful in this assignment. Here are some links to the documentation for a few of these methods:

1. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html#contains(java.lang.CharSequence)
2. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html#format(java.lang.String,java.lang.Object...)
3. https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html#toLowerCase()