This announcement is an invitation to apply to participate in a workshop
which is part of the Usability Professionals Association 2001 Conference
in Las Vegas, Nevada, June 25 to 29, 2001 (see
http://www.upassoc.org/conf2001/reg/ for details).
INVITATION to Submit POSITION PAPERS to UPA 2001 Workshop #6:
Exploring Measurement and Evaluation Methods For Accessibility
Sharon Laskowski, Mark Stimson, and I would like to invite both
usability researchers and practioners to submit 1-3 page position papers
to our workshop where we hope to foster discussion that will lead to the
development of methods for measuring and reporting on the accessibility
of electronic and information technology (E&IT) products by people with
If you have any questions, please feel free to email me at
[log in to unmask] and [log in to unmask]
The position paper deadline is May 11, 2001. Please consult the UPA
conference website for additional details on how to register for the
Thanks! Hope to see you in June! Bill, Sharon, and Mark
For your convenience, here is the workshop description from the UPA
Exploring Measurement and Evaluation Methods For Accessibility
Bill LaPlant, U.S. Census Bureau
Sharon Laskowski, National Institute of Standards and Technology
Mark Stimson, Oracle Corporation
Tuesday, June 26, 8:30 a.m. to 5:30 p.m.
Audience: Intermediate and Advanced Usability Specialists
Keywords: Metrics, possible new method, disability
The objective of this workshop is to foster discussion that will lead to
the development of methods for measuring and reporting on the usability
of electronic and information technology (E&IT) products for the access
of data and information by people with disabilities. Improved usability
for this group of users will not only help address regulatory
requirements, but also encourage improved product design in general. The
hope is that new or adapted usability evaluation methods for measuring
professionals to provide the sound advice needed by industry product
developers, service providers, as well as procurement officials to
ensure accessibility of E&IT products.
There is increased interest in accessibility because of recent United
States government legislation. Both the Americans with Disabilities Act
of 1990 and the Rehabilitation Act of 1973 have established the right of
individuals with disabilities to be provided accommodations by
businesses that deal with the public and in the work place. Section 508
of the Rehabilitation Act was amended by passage of the Workforce
Investment Act of 1998 to require that Federal agencies' electronic and
information technology be accessible to people with disabilities. Part
of this requirement is to ensure that "individuals with disabilities who
are Federal employees (or members of the general public) have access to
and use of information and data that is comparable to the access to and
use of the information and data by Federal employees (and members of the
general public) who are not individuals with disabilities." Federal
agencies is are to meet this requirement by acquiring only E&IT that
comply with accessibility standards developed by the U.S. Architectural
and Transportation Access Compliance Board (the "Access Board").
The Section 508 standards provide no formal guidance to industry or
procurement officials concerning how to determine compliance. However,
because the standards are to be implemented as a regulatory mandate for
enforcing Section 508 accessibility, they must be unambiguous and
testable. To further support this, the Federal government has funded a
pilot project to establish a testing program for evaluating E&IT
compliance and interoperability and developing industry consensus
standards for accessibility called ADIT(Accessibility for People with
Disabilities through Standards, Interoperability, and Testing). However,
it is our opinion that testing to standards alone without user testing
will not necessarily address the issue of comparable access by the
Measuring comparable access means that one must measure the success of
the user interaction with the product. In determining the usability of a
product by a disabled population, one must apply metrics of efficiency,
effectiveness, and satisfaction--the same metrics that the usability
engineering community employs in a typical usability evaluation. At a
low level, some evaluation can be done by inspection, for example, by
applying guidelines such as checking for meaningful alternative text for
images on a web page. However, performance-based metrics are the only
ones that can, with reasonable certainty, determine whether users with a
certain type of disability can perform their job with the product in a
comparable way to users without that disability.
The difficult challenge that must be met by the usability community is
to clearly articulate how products can be designed and tested for
accessibility and to develop techniques to accomplish this. In
particular, low cost usability test processes that measure, in a
repeatable and reproducible way, the user performance at the human
computer interface are needed.
What specific issues do we want to address?
The high level issue we are exploring is how to effectively test for
accessibility: What usability techniques can be applied and can they be
executed quickly at relatively low cost? For example, is it possible to
develop repeatable, reproducible metrics, and associated test
methodology for measuring the usability of supporting the access of data
and information by people with disabilities? Given such a methodology,
one could determine acceptable performance levels for assistive
technology and develop accessibility reporting mechanisms to support
conformance testing and certification for Section 508. To accomplish
this we believe that there are four issues that need to be addressed by
the usability community (and, we hope, by this workshop):
* Qualification of test subjects: If a given product must be shown to be
> accessible by the entire range of people with disabilities, the claim
> is that there are over 160 groups of users that would have to be
> tested (private communication, Gregg Vanderheiden, TRACE R&D Center).
> It might be possible to combine groups so that the number of test
> groups could be as low as 5 or 6. This would require peer-reviewed
> research before the results could be considered scientifically valid,
* Development of "standard" tasks: A set of standard tasks is needed to
> ensure that the tests for comparable access are reproducible. What are
> these tasks? How should they be specified?
* Development of "standard" performance measures: The metrics and
> measurements applied to evaluate the performance of a set of subjects
> on a task must also be standardized so that results of multiple tests
> can be compared. A metric for comparability is also important. What
> are the metrics? What are some candidate measurement tools?
* Agreement on a "standard" reporting mechanism: A standard reporting
> mechanism is essential for establishing comparability and
> reproducibility of the results. Can a modified version of the Industry
> Usability Reporting Workshop's Common Industry Format (IUSR's CIF --
> See http://www.nist.gov/iusr ) serve as a reasonable basis for such a
> mechanism? What are the alternatives?
An associated issue is that of cost. If it is possible to create a
minimal set of tests based on optimizing the number of test subjects,
standard tasks and measures, the cost might be within acceptable limits.
We also encourage discussion of how test and evaluation methods that do
not necessarily require user testing can be adapted for various stages
of the design and development process to address accessibility.
Who should participate in the workshop?
We invite position papers from usability specialists who have experience
in testing with disabled users, universal design, or usability
evaluation for heterogeneous user populations. Researchers who have
experience in developing or adapting methodologies for evaluation are
also encouraged to submit position papers.
Registering for a Workshop
Workshops are closed sessions; you may register for a workshop only with
permission of the workshop organizers. To participate, send your
position paper to Bill LaPlant at [log in to unmask] and
[log in to unmask] by May 11, 2001. Electronic applications may be
submitted in plain text (.txt), rich text format (.rtf), or Microsoft
Word for Windows.
Your registration should include the workshop you are requesting.
Workshop registration will be processed when approved by the workshop
coordinator. Workshop selections should be complete by May 22, 2001.
What do we want people to cover in their position papers?
Potential participants are asked to submit a 1-3 page position paper
briefly describing the evaluation or design work they have done,
including experience with any special user populations. The paper should
also include a short description of at least 3 issues they have
encountered (or think they might encounter) in doing their evaluations
and how they pertain that to the issues described above. We will collate
these from accepted participants' submissions and distribute them to the
group prior to the workshop. Participants will be expected to read these
issues and the position papers prior to the workshop.
How do we expect the day to go?
The facilitators will present the issues in terms of the requirements
of the accessibility regulations and the impact on the usability testing
and evaluation process. Workshop participants will be asked to
participate in exercises designed to develop pragmatic approaches to the
workshop issues. Workshop participants should be prepared to work with
the facilitators on a framework for developing a process for evaluation
for accessibility. This will include an extension to the Common Industry
Format or alternatives for reporting on testing.
One or more of the facilitators have investigated the implications of
Section 508 and testing for accessibility, have been involved in the
development or use of the CIF, or have experience in usability testing
using people with disabilities. They will work with the appropriate
groups to ensure that the discussions stay focused, as time for subgroup
discussion is limited. The facilitators have also arranged for a member
of the ADIT team to attend (see: http://adit.aticorp.org/) and help with
the production of the final workshop report.
What deliverables will come out of the workshop?
The workshop leaders will produce a report from the workshop suitable
for submission to User Experience in order to share the insights that
come from the workshop with the UPA membership. Results of the workshop
will be provided to workshop participants and to UPA 2002 through
publications and web site materials. The workshop results will also be
presented to the ADIT Forum (see:
http://adit.aticorp.org/accessibility-forum/reg-announcement.html) as a
starting point for their consideration for testing methods for
determining comparable access of E&IT by people with disabilities.
William P. LaPlant, Jr. Census; Rm.3000-4; Washington DC 20233
Computer Scientist Phone:301-457-4887 Home:703-360-9184
Technology Research Staff mailto:[log in to unmask]
-- mailto:[log in to unmask]
I am committed to Children inheriting a culture of unlimited
possibilities; Technology empowering miraculous lives.