Please find enclosed to this message the Call for Paper for the Workshop
"*Review, Report and Refine Usability Evaluation Methods (R3 UEMs)"*
to be held in Athens, Greece on 5th March 2007.

Apologizes for cross posting.


Review, Report and Refine Usability Evaluation Methods (R3 UEMs)



Dominique Scapin

Institut National de Recherche en Informatique et en Automatique (INRIA)


Email: [log in to unmask]


Effie Lai-Chong Law

Computer Engineering and Networks Laboratory (TIK)

ETH Zürich


Email: [log in to unmask]



A full day workshop will be held under the auspices of COST294-MAUSE 

Date:             5th March 2007 (Monday)

Location:         Athens, Greece




There exist a variety of usability evaluation methods (UEMs), which are 
employed in a wide spectrum of contexts by people with different 
backgrounds, goals and needs.  Selecting appropriate UEMs to meet 
contextual requirements and constraints is the foremost and crucial step 
that leads to useful evaluation outcomes and presumably effective 
redesign of the system of interest.  Furthermore, emerging information 
technologies (IT) such as ambient intelligence, pervasive computing and 
universal accessibility have triggered the development of new evaluation 
methods and tools, which have been adopted or trialled by a local 
research group but not yet well disseminated to the entire usability 
community. A refined and consolidated knowledge-pool about established 
as well as emerging UEMs based on expertise and experiences of usability 
practitioners and researchers is deemed desirable. It will not only 
enable the selection of right methods but also serve as valuable 
resources for informing experienced members of the usability community 
about new UEMs as well as for training newcomers about the development 
of UEMs.  With the aim to build such a knowledge-pool, the WG1 (working 
group) of the project COST294-MAUSE has undertaken the challenge to 
develop an approach to critically reviewing and analyzing UEMs.  


Specifically, WG1 has developed several instruments:

(i) A classification scheme of UEMs

Three major categories are DGMM (Data Gathering & Modelling Methods); 
UIEM (User Interactions Evaluation Methods); CMs (Collaborative 
Methods), each of which is further divided into sub-categories.

(ii) Two templates:

*     «Generic Methods» - to support descriptions of widely used UEMs at 
the generic level, i.e. mainly using reference material such as 
publication, courses, etc.

*     «Case Studies» - to support description of actual cases of UEM 
implementation, i.e. details on how a specific method was used, with its 
context, its precise usage of the method.

(iii) A guidance document for these templates.


Selected UEMs are categorized, critically reviewed and analyzed on 
different aspects, from bibliographical references to 
advantages/disadvantages, through a set of methodological attributes. 
Individual reviews are documented as a set of records in the MAUSE 
Digital Library.  Best practices of existing UEMs, covering operational, 
organizational and cultural dimensions, can be derived from these 
records and rendered accessible to the usability community.


Up to now, a number of UEMs have systematically been reviewed by some 
COST294-MAUSE partners with the use of the scheme and templates 
described above, including:

* Cognitive Walkthrough

* CASSM (Concept-based Analysis of Surface and Structural Misfits)

* Ergonomic Criteria

* K-MADe (Kernel of Model for Activity Description environment)

* MOT (Metaphors Of human Thinking)

* CUT (Cooperative Usability Testing)

* Personas

* Heuristic Evaluation

* User Performance Testing

* EU-CON II Evaluation

* Abstract Task Inspection

* A set of modelling methods: GOMS, KLM, NGOMSL, TAG, HTA, TKS, GTA and CTT

These reviews were compiled (August 2006) into a document entitled 
«Usability Evaluation Methods Classification, Description and Template: 
COST294-MAUSE WG1 2nd Interim Report» (hereafter - UEM Reviews).


To further enrich the scope and quality of this knowledge-pool, we aim 
to invite more contributions from the entire usability community.  Case 
studies from industry are particularly welcome.



 The workshop R3 UEMs aims to achieve four major goals:

* To invite more systematic, critical reviews on a variety of UEMs, 
including those listed above and many others;

* To perform meta-review of existing reviews listed above;

* To derive or refine best practices of established UEMs;

* To explore emerging UEMs, identifying their applicability and 



30: Usability researchers and practitioners, including:

* Contributors who have already submitted reviews on UEMs (see «UEM 

* Contributors who will submit reviews on UEMs that have not been yet 

* Meta-reviewers of existing reviews documented in the report;

* Creators of emerging UEMs;



Two major types of contributions are:


(a) Review of a UEM not currently covered (i.e., complete a template for 
a UEM not currently covered in «UEM Reviews»)

* If you are interested in participating, please send an expression of 
interest to the Chair and Co-chair of the Workshop who will provide 
access to draft documents and templates for new method coverage

* To participate, you would need to complete a template for your 
selected UEM


(b)  Meta-review of UEM described

* If you are interested in participating, please send an expression of 
interest to the Chair and Co-chair of the Workshop who will provide 
access to draft documents and templates for new method coverage and 
reviews of existing methods

* To participate, you would need to complete at least three meta-reviews 
of current method descriptions in «UEM Reviews»






15th January 2007:      Deadline for submission

31st January 2007:      Authors of accepted contributions notified

15th Feb 2007:    Deadline for application for sponsorship1


All submissions will be assessed on the relevance to the Workshop. It is 
expected that at least one of the authors of each contribution will 
attend the workshop.



Prior to the Workshop, accepted contributions will be integrated into an 
enlarged and revised version of the report «UEM Reviews», which will be 
distributed to the Workshop's participants. A questionnaire will also be 
administered to collect the participants' opinions, comments and 
questions about the report.  The data thus collected will be 
consolidated and addressed in the Workshop. 


On the day of the Workshop, the following activities will be conducted:

(i) Invited Talk (~ 1 hour)

An expert will be invited to present a talk on revision of ISO standards 
and their impacts on industry work


(ii) Presentations of emerging UEMs (~ 2 hours):

Quality contributions that review emerging UEMs will be selected for 
presentation; 15 minutes are allocated for each emerging UEM. By 
emerging UEMs, we refer to those methods which have recently been 
developed (i.e. less than ten-year old), applied and validated, but are 
not yet widely used in the usability community and not yet described in 
the current WG1 interim report.


(iii) Group Discussions (~ 3 hours):

      Contributors who have reviewed or meta-reviewed a specific UEM 
will form a working group (i.e. approx. 5 members) to discuss various 
aspects of the UEM:

* Scoping

* Strengths and weaknesses

* Refinement suggestions

* Best practices and recommendation

* Proposed changes to the current review templates and review procedures

Depending on the number and nature of contributions, parallel sessions 
on different UEMs may be organized.


(iv) Plenary Reporting and Forum (~ 1 hour):

      Each working group is to present their findings (~ 10 minutes each 
with follow-up questions from the audience)



Authors of quality contributions will be invited to engage in follow-on 
activities listed below:

* A tutorial of usability evaluation methods;

* A practical handbook/guidebook on usability evaluation methods;

* An open library of usability evaluation methods;

* A special issue in an HCI journal on methodologies of usability 

Proposals on other follow-on activities are welcome.



UI designers, usability researchers and practitioners, and advanced 
postgraduates in HCI





Interested participants are required to use the given templates (Word 
file) to write up your contributions, which are to be submitted as an 
attachment of an email to:

Chair (Dominique Scapin: [log in to unmask]) and

Co-chair (Effie Law: [log in to unmask]). 

Further enquiries can be sent to Chair/Co-Chair.

1 For non-COST294 members: A fixed-rate or a full sponsorship covering 
travel and accommodation will be granted, depending on the number of 
eligible applicants. 


                To unsubscribe, send an empty email to
     mailto:[log in to unmask]
    For further details of CHI lists see