ACM SIGCHI General Interest Announcements (Mailing List)


Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Condense Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
"ACM SIGCHI General Interest Announcements (Mailing List)" <[log in to unmask]>
Anca Dumitrache <[log in to unmask]>
Fri, 4 Jan 2019 10:19:55 +0100
text/plain; charset="UTF-8"
Anca Dumitrache <[log in to unmask]>
text/plain (190 lines)

Subjectivity, Ambiguity and Disagreement in Crowdsourcing

2nd International Workshop on Subjectivity, Ambiguity and Disagreement in
Crowdsourcing Workshop, co-located with the International Web Conference


>>>>>> IMPORTANT DATES >>>>>>

Abstract submission: 20 Jan 2019

Paper submission 1 Feb 2019
Author notification: 24 Feb 2019
Final version deadline: 3 Mar 2019
Workshop date: 13/14 May 2019

>>>>>> CALL FOR PAPERS >>>>>>

Ambiguity creates uncertainty in practically every facet of the web.
Machine learning, recommender systems, web search, as well as news and
systems that support human discourse, experience aspects of ambiguity and
subjectivity that leads to inaccuracies, poor and undesirable performance.
In crowdsourcing, human computation, and human-in-the-loop systems, topics
central to TheWebConf, recent research has attempted to delve deeper into
subjectivity and ambiguity, finding it to manifest in human-collected data
primarily as disagreement [Cheatham and Hitzler, 2014; Plank, Hovy and
Sogaard, 2014; Bayerl and Paul, 2011; Aroyo and Welty, 2015; Schaekermann,
Law, Williams and Callaghan, 2016; Chang, Amershi and Kamar, 2017; Lin and
Weld, 2014; etc.]. This includes the information presented to workers as
part of a crowdsourcing task, the instructions for what to do with it, and
the information they are asked to provide. These ambiguities become deeply
tied into our machine learning models and metrics, as they are in the gold
data. Similar ambiguity is found in interpreting and deriving utility from
user generated data from large scale systems such as social media and
search engines. Another aspect of disagreement surfaces in collaborative
projects such as Wikipedia, online forums, and semantic markups. In
language, ambiguity can result from missing details, contradictions and
subjectivity. Subjectivity may stem from differences in cultural context,
life experiences, or individual perception of hard-to-quantify properties.
All of these can leave people with conflicting interpretations, leading to
results that system builders would regard as "wrong". These issues share
common ground with many areas of interest to TheWebConf.

SAD2019 (Subjectivity, Ambiguity and Disagreement in Crowdsourcing)
Workshop aims to bring together a latent community of researchers who treat
disagreement (and subjectivity and ambiguity) as signal, rather than noise.
Such researchers use theoretical and empirical methodology to characterize,
utilize, mitigate and derive value from subjectivity, ambiguity and
disagreement.  The workshop will include invited talks, short technical
talks and a discussion of medium- and long-term challenges to fuel future

We encourage interdisciplinary submissions from the broad spectrum of
and human computation research in fields and application areas, such as
computer science, information sciences, law, medical data analysis,
communication science and political science, as well as those primarily
working on human computation and crowdsourcing.  Solutions to these
challenging problems will benefit from a diverse set of perspectives.
Topics of interest (but not limited to) are:


   Interaction/relation between disagreement, ambiguity and subjectivity

   Costs and challenges introduced by ambiguity

   Designing tasks with high subjectivity and low inter-rater reliability
   (e.g., semantic, linguistic, common sense, moral judgements.)

   Better metrics for characterizing disagreement (over traditional
   inter-rater reliability)

   Ambiguity in human computation task design, how to identify it and what
   to do about it

   Theoretical ambiguity-aware frameworks for collecting data

   Teasing apart different sources of ambiguity

   Best practices for collecting subjective data

   Benchmarks and datasets for studying ambiguity

   Grand challenges that will further our understanding of subjectivity,
   ambiguity and disagreement

   Disagreement, ambiguity, subjectivity and their representation and
   interaction in different content modalities, e.g. text, images, videos,

   Interdisciplinary perspectives on disagreement, ambiguity and
   subjectivity, e.g., law, political science, humanities

>>>>>> SUBMISSION >>>>>>

Authors can submit four types of papers:


   short papers (up to 6 pages in length), plus unlimited pages for

   full papers (up to 10 pages in length), plus unlimited pages for

   position papers (up to 4 pages in length), plus unlimited pages for

   demo papers (up to 4 pages in length), plus unlimited pages for

Submissions may present ideas, results from experimental studies,
methodologies, work-in-progress and/or applications of systems that explore
the topics of subjectivity, ambiguity and disagreement  in crowdsourcing
and human computation in relation to computational systems, applications,
or services. Page limits include diagrams and appendices. Submissions
should be formatted according to the formatting instructions in the General
proceedings of the workshop will be published jointly with the conference
proceedings of TheWebConf 2019.

Submit papers through

All submissions must be written in English.

>>>>>> WORKSHOP CHAIRS >>>>>>

Anca Dumitrache, Vrije Universiteit Amsterdam, [log in to unmask]

Alex Quinn, Purdue University, [log in to unmask]

Olivia Rhinehart, Jigsaw, [log in to unmask]

Mike Schaekermann, University of Waterloo, [log in to unmask]

Michael Tseng, Google, [log in to unmask]

Chris Welty, Google, [log in to unmask]

Please let us know if you are interested in contributing in the workshop
and which of the possible categories. For more submission details, please
consult the workshop website:

Anca Dumitrache
PhD student, User-Centric Data Science
Computer Science Department
Vrije Universiteit Amsterdam

Twitter: @anca_dmtrch

    For news of CHI books, courses & software, join CHI-RESOURCES
     mailto: [log in to unmask]

    To unsubscribe from CHI-ANNOUNCEMENTS send an email to
     mailto:[log in to unmask]

    For further details of CHI lists see