[QA] FiveUI: automate testing for accessibility

Tue Jul 9 16:16:18 UTC 2013

On Tue, Jul 9, 2013 at 6:48 AM, Željko Filipin <zfilipin at wikimedia.org>wrote:

> On Wed, Jul 3, 2013 at 11:49 PM, Sumana Harihareswara <
> sumanah at wikimedia.org> wrote:
>
>> It seemed to me like it might play well with our current automated
>> testing regime but I wanted to spread the word to this list to ask for
>> others' opinions.
>>
>
> I have recently read about something similar:
>
>
> http://watirmelon.com/2013/02/12/automated-local-accessibility-testing-using-wave-and-webdriver/
>
> Chris, what do you think? Is that something we should do these days?
>
>
I would really hesitate to go very far down this road without a mandate
from the Platform team and without people available to implement changes
for issues found with accessibility.  As Timo mentioned in San Francisco at
the Wikilove automation training, pages on Wikipedia today do not even
comply with W3C standards.

>From doing browser automation against a wide range of existing features, I
can say that accessibility from feature to feature varies widely, with no
feature being particularly accessible, and many being downright awful.

Let's look at one example.  Here is a discussion of why having a :title
attribute for links is far superior to having only link text available for
links. http://www.jimthatcher.com/news-061607.htm  There are also
implications for localization.

Below is a list of links, divs, and spans (mostly links) in our automated
tests that have only link text available as an identifier:

$ grep -r 'text:' *
features/support/modules/interlanguage_module.rb:  a(:back_to_display,
text: 'Back to display settings')
features/support/modules/interlanguage_module.rb:  a(:back_to_input, text:
'Back to input settings')
features/support/modules/interlanguage_module.rb:  a(:english_link, text:
'English')
features/support/modules/interlanguage_module.rb:  a(:talk, text:
'Discussion')
features/support/pages/upload_page.rb:  span(:continue, text: 'Continue')
features/support/pages/random_page.rb:  a(:create_a_book, text: 'Create a
book')
features/support/pages/random_page.rb:  a(:download_as_pdf, text: 'Download
as PDF')
features/support/pages/random_page.rb:  a(:download_the_file, text:
'Download the file')
features/support/pages/random_page.rb:  a(:print_export, text:
'Print/export')
features/support/pages/random_page.rb:  a(:printable_version, text:
'Printable version')
features/support/pages/release_rights_page.rb:
 page.next_parent_element.span_element(text: 'Next')
features/support/pages/describe_page.rb:
 page.next_parent_element.span_element(text: 'Next')
features/support/pages/preferences_appearance_page.rb:  a(:css_link,
text:'Custom CSS')
features/support/pages/preferences_appearance_page.rb:  a(:javascript_link,
text: 'Custom JavaScript')
features/support/pages/preferences_appearance_page.rb:
 a(:restore_default_link, text:'Restore all default settings')
features/support/pages/preferences_appearance_page.rb:
 button(:save_button, text: 'Save')
features/support/pages/use_page.rb:  span(:upload_more_files, text: 'Upload
more files')
features/support/pages/visual_editor_page.rb:  a(:edit_ve, text: 'Edit')
features/support/pages/visual_editor_page.rb:  span(:looks_good, class:
've-ui-labeledElement-label', text: 'Looks good to me')
features/support/pages/visual_editor_page.rb:  span(:return_to_save, class:
've-ui-labeledElement-label', text: 'Return to save form')
features/support/pages/visual_editor_page.rb:  span(:review_changes, class:
've-ui-labeledElement-label', text: 'Review your changes')
features/support/pages/visual_editor_page.rb:  span(:save_page, class:
've-ui-labeledElement-label', text: 'Save page')
features/support/pages/visual_editor_page.rb:  span(:second_save_page,
class: 've-ui-labeledElement-label', text: 'Save page', index: 1)
features/support/pages/visual_editor_page.rb:    span(:add_parameter, text:
'Add parameter', frame: frame)
features/support/pages/visual_editor_page.rb:    span(:add_template, text:
'Add template', frame: frame)
features/support/pages/visual_editor_page.rb:    span(:apply_changes, text:
'Apply changes', frame: frame)
features/support/pages/visual_editor_page.rb:    span(:create_new,
text:'Create new source', frame: frame)
features/support/pages/visual_editor_page.rb:    span(:remove_parameter,
text: 'Remove parameter', frame: frame)
features/support/pages/visual_editor_page.rb:    span(:remove_template,
text: 'Remove template', frame: frame)
features/support/pages/visual_editor_page.rb:
 list_item(:template_list_item, text: 'S', frame: frame)
features/support/pages/learn_page.rb:
 page.next_parent_element.span_element(text: 'Next')
features/support/pages/aftv5_page.rb:  a(:all_comments, text: /All
comments/)
features/support/pages/aftv5_page.rb:  a(:back_to_yesno, text: 'Back to
step 1')
features/support/pages/aftv5_page.rb:  span(:create_account, text: 'Create
an account')
features/support/pages/aftv5_page.rb:  a(:edit, text: 'Edit')
features/support/pages/aftv5_page.rb:  a(:feedback_page, text: 'feedback
page')
features/support/pages/aftv5_page.rb:  a(:helpful_feedback, href: '
http://en.wikipedia.org/wiki/Wikipedia:Feedback_guidelines', text: 'helpful
feedback')
features/support/pages/aftv5_page.rb:  a(:learn_more, text: /Learn more/)
features/support/pages/aftv5_page.rb:  a(:log_in, text: 'Log in')
features/support/pages/aftv5_page.rb:  a(:most_relevant, text: /Most
relevant/)
features/support/pages/aftv5_page.rb:  span(:no, text: 'No')
features/support/pages/aftv5_page.rb:  span(:post_feedback, text: 'Post
your feedback')
features/support/pages/aftv5_page.rb:  a(:terms, href: '//
wikimediafoundation.org/wiki/Feedback_privacy_statement', text: 'terms')
features/support/pages/aftv5_page.rb:  a(:view_article, text: 'View
article')
features/support/pages/aftv5_page.rb:  a(:whats_this, text: "What's this?")
features/support/pages/aftv5_page.rb:  span(:yes, text: 'Yes')
features/support/pages/edit_page.rb:  a(:advanced, text: 'Advanced')
features/support/pages/edit_page.rb:  a(:bold, text: 'Bold')
features/support/pages/edit_page.rb:  a(:edit, text: 'Edit')
features/support/pages/edit_page.rb:  a(:embedded, text: 'Embedded file')
features/support/pages/edit_page.rb:  a(:italic, text: 'Italic')
features/support/pages/edit_page.rb:  a(:link, text: 'Link')
features/support/pages/edit_page.rb:  a(:read, text: 'Read')
features/support/pages/edit_page.rb:  a(:reference, text: 'Reference')
features/support/pages/edit_page.rb:  a(:sig, text: 'Signature and
timestamp')
features/support/pages/edit_page.rb:  a(:special, text: 'Special
characters')
features/support/pages/edit_page.rb:  a(:view_history, text: 'View history')
features/support/pages/preferences_page.rb:  a(:appearance_link, text:
'Appearance')
features/support/pages/preferences_page.rb:  a(:date_and_time_link, text:
'Date and time')
features/support/pages/preferences_page.rb:  a(:editing_link, text:
'Editing')
features/support/pages/preferences_page.rb:  a(:gadgets_link, text:
'Gadgets')
features/support/pages/preferences_page.rb:  a(:misc_link, text: 'Misc')
features/support/pages/preferences_page.rb:  a(:pending_changes_link, text:
'Pending changes')
features/support/pages/preferences_page.rb:  a(:recent_changes_link, text:
'Recent changes')
features/support/pages/preferences_page.rb:  a(:search_link, text: 'Search')
features/support/pages/preferences_page.rb:  a(:user_profile_link, text:
'User profile')
features/support/pages/preferences_page.rb:  a(:watchlist_link, text:
'Watchlist')
features/support/pages/wikilove_page.rb:  div(:barnstars, text: 'Barnstars')
features/support/pages/wikilove_page.rb:  div(:food_and_drink, text: 'Food
and drink')
features/support/pages/wikilove_page.rb:  a(:heart, text: 'WikiLove')
features/support/pages/wikilove_page.rb:  div(:kittens, text: 'Kittens')
features/support/pages/tour_page.rb:  a(:edit, text: 'Edit')
features/support/pages/tour_page.rb:  a(:next, text: 'Next')
features/support/pages/move_page.rb:    @browser.a(text: text)
features/support/pages/move_page.rb:    @browser.a(text: "#{text} Moved")
features/support/pages/login_page.rb:  a(:password_strength, text:
'password strength')
features/support/pages/login_page.rb:  a(:phishing, text: 'phishing')
features/support/pages/main_page.rb:  a(:login, text: 'Log in')
features/support/pages/main_page.rb:  a(:logout, text: 'Log out')
features/support/pages/main_page.rb:  a(:my_sandbox, text: 'My sandbox')
features/support/pages/main_page.rb:  a(:preferences, text: 'Preferences')
features/support/pages/main_page.rb:  a(:talk, text: 'Talk')
features/support/pages/upload_wizard_page.rb:  a(:categories, text: 'Add
categories and more information ...')
features/support/pages/upload_wizard_page.rb:  a(:cc, text: 'The copyright
holder published this work with the right Creative Commons license')
features/support/pages/upload_wizard_page.rb:  a(:different_license, text:
'Use a different license')
features/support/pages/upload_wizard_page.rb:  a(:expired, text: 'The
copyright has definitely expired in the USA')
features/support/pages/upload_wizard_page.rb:  a(:flickr, text: 'The
copyright holder published their photo or video on Flickr with the right
license')
features/support/pages/upload_wizard_page.rb:  a(:found_it, text: "I found
it on the Internet -- I'm not sure")
features/support/pages/upload_wizard_page.rb:  a(:logged_in, text: 'logged
in')
features/support/pages/upload_wizard_page.rb:  span(:next, text: 'Next')
features/support/pages/upload_wizard_page.rb:  a(:not_mentioned, text:
'Another reason not mentioned above')
features/support/pages/upload_wizard_page.rb:  a(:recommended_license,
text: 'Use the recommended license')
features/support/pages/upload_wizard_page.rb:  a(:us_govt, text: 'This work
was made by the United States government')
features/support/pages/article_page.rb:  a(:discussion, text: 'Discussion')
features/support/pages/article_page.rb:  a(:edit, text: 'Edit')
features/support/pages/article_page.rb:  a(:move_link, text: 'Move')
features/support/pages/page_triage_page.rb:  a(:learn_more, href:
/Wikipedia:Page_Curation\/Help/, text: 'Learn more')
features/support/pages/page_triage_page.rb:  a(:leave_feedback, href:
/Wikipedia_talk:Page_Curation/, text: 'Leave feedback')
features/support/pages/page_triage_page.rb:  a(:review, text: 'Review')
features/support/pages/does_not_exist_page.rb:  a(:create_source, text:
'Create source')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20130709/e0c7db23/attachment.html>