ACM SIGCHI General Interest Announcements (Mailing List)


Options: Use Forum View

Use Monospaced Font
Show HTML Part by Default
Condense Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
"ACM SIGCHI General Interest Announcements (Mailing List)" <[log in to unmask]>
Fri, 15 Mar 2019 13:30:36 +0000
Liadh Kelly <[log in to unmask]>
text/plain; charset="UTF-8"
Liadh Kelly <[log in to unmask]>
text/plain (132 lines)
======================== CALL FOR PAPERS =======================

*EvalUMAP 2019 Workshop*

*Towards comparative evaluation in user modeling, adaptation and

* <>  *


To be held in conjunction with the 27th Conference on User Modeling,
Adaptation and Personalization, UMAP 2019, June 2019, Cyprus

**Call for critiques of proposed shared evaluation challenge approach
(available at:**
**Call for Papers**

*Max paper length:*  2-pages

*Submission deadline:* March 20, 2019

Research in the areas of User Modelling, Adaptation and Personalization
faces a number of significant scientific challenges. One of the most
significant of these challenges is the issue of comparative evaluation. It
has always been difficult to rigorously compare different approaches to
personalization, as the function of the resulting systems is, by their
nature, heavily influenced by the behavior of the users involved in
trialing the systems. Developing comparative evaluations in this space
would be a huge advancement as it would enable shared comparison across

We envision that EvalUMAP 2019 will be the starting point for an annual
comparative evaluation challenge at future UMAP conferences.  The two
previous iterations of this workshop (EvalUMAP 2016 and 2017) have created
a platform for this topic to be discussed and advanced. In this iteration
of the workshop series, final steps towards shared challenge generation
will be made. A white paper mapping a proposed shared challenge approach
has been distributed to the community (available at:, consisting of exemplar
data, desirable metrics and a plan for what engaging in the challenge task
would look like. Feedback in the form of reflection, critique and alternate
approaches are solicited from the community via short submissions to the
EvalUMAP workshop.

The planned outcome of the workshop will be a roadmap to develop initial
shared task(s) that will be published ahead of UMAP 2020, providing
opportunity for participants to test and tune their systems and complete
the task in order for comparative results and associated publications to be
prepared for and presented at UMAP 2020. The resulting shared task(s) will
be accompanied by appropriate models, content, metadata, user behaviors,
etc., and can be used to comprehensively compare how different approaches
and systems perform. In addition, a number of metrics and observations for
participants to perform in order to facilitate comparison will be outlined.

Workshop topics are evaluation focused and include, but are not limited to:

- Reflecting upon and critiquing the proposed shared challenge approach
white paper (
- Understanding UMAP evaluation
- Defining tasks and scenarios for evaluation purposes
- Identification of potential corpora for shared tasks
- Interesting target tasks and explanations of their importance
- Critiques or comparisons of existing evaluation metrics and methods
- How we can combine existing evaluation metrics and methods
- Improving on previously suggested metrics and methods
- Reducing the cost of evaluation
- Proposal of new evaluation metrics and methods
- Technical challenges associated with design and implementation
- Privacy, Ethics and security issues
- Legal and ethical issues

*Workshop format:*

This will be an interactive workshop structured to encourage group
discussion and active collaboration among attendees.

*Paper Submissions:*

The workshop is now accepting paper submissions. Position papers (max 2
pages) either critiquing our shared challenge proposal white paper
(available at:, proposing an
alternate shared challenge approach, or reflecting on experiences on the
topics of the workshop are invited. Submissions should be in ACM SIGS
format. Templates are available at

Papers should be submitted in pdf format through the EasyChair system no later than midnight
Pacific Daylight Time on March 20, 2019. Submissions will be reviewed by
members of the workshop program committee. Accepted papers will be included
in the UMAP 2019 EvalUMAP workshop proceedings.

*Important Dates:*

March 20, 2019: Deadline for paper submission (midnight Pacific Daylight
March 27, 2019: Notification to authors
April 3, 2019: Camera-ready paper due
June, 2019: Workshop during UMAP

*Further Information:*

Further information is available on the workshop website at or by emailing [log in to unmask]

*Workshop Chairs:*

Liadh Kelly, Maynooth University, Ireland
Bilal Yousuf, Trinity College Dublin, Ireland

*Workshop Co-organizers:*

Owen Conlan, Trinity College Dublin, Ireland
Kieran Fraser, Trinity College Dublin, Ireland
Kevin Koidl, Trinity College Dublin, Ireland
Séamus Lawless, Trinity College Dublin, Ireland

    For news of CHI books, courses & software, join CHI-RESOURCES
     mailto: [log in to unmask]

    To unsubscribe from CHI-ANNOUNCEMENTS send an email to
     mailto:[log in to unmask]

    For further details of CHI lists see