CHI-ANNOUNCEMENTS Archives

ACM SIGCHI General Interest Announcements (Mailing List)

CHI-ANNOUNCEMENTS@LISTSERV.ACM.ORG

Options: Use Classic View

Use Monospaced Font
Show HTML Part by Default
Show All Mail Headers

Topic: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Aleksandr Chuklin <[log in to unmask]>
Wed, 15 Jul 2020 13:25:36 +0200
text/plain (149 lines)
******************************************************************************************************

                                                CALL FOR CHALLENGE PARTICIPATION

******************************************************************************************************

ClariQ (pronounced as Claire-ee-que) challenge is organized as part of
the Conversational AI challenge series (ConvAI3) at Search-oriented
Conversational AI (SCAI) EMNLP workshop in 2020 (https://scai.info).
The main aim of the conversational systems is to return an appropriate
answer in response to the user requests. However, some user requests
might be ambiguous. In IR settings such a situation is handled mainly
through the diversification of a search result page. It is however
much more challenging in dialogue settings. Hence, we aim to study the
following situation for dialogue settings:

  - a user is asking an ambiguous question (a question to which one
can return > 1 possible answers);
  - the system must identify that the question is ambiguous, and,
instead of trying to answer it directly, ask a good clarifying
question.

The main research questions we aim to answer as part of the challenge
are the following:

  - RQ1: When to ask clarifying questions during dialogues?
  - RQ2: How to generate the clarifying questions?

******************************************************************************************************

  Web: http://convai.io
  Email for questions: [log in to unmask]
  Twitter: https://twitter.com/scai_workshop


*****************************************************************************************************
                                                 HOW TO PARTICIPATE
******************************************************************************************************

  - In order to get to the leaderboard please register your team here:
https://docs.google.com/forms/d/e/1FAIpQLSer8lvNvtt-SBwEtqZKjMtPJRWmw5zHUxoNgRJntzBIuVXrmw/viewform
  - The datasets, baselines and evaluation scripts for the Stage 1 are
available in the following repository
https://github.com/aliannejadi/ClariQ
  - Email us if you have any questions

*****************************************************************************************************
                                                 IMPORTANT DATES
******************************************************************************************************
The challenge will be run in two stages:
  - Stage 1: fixed dataset
  - Stage 2: human-in-the-loop evaluation

Timeline:
  - Stage 1 will take place from **July 7, 2020 – September 9, 2020**.
Up until September 9, 2020 participants will be able to submit their
models (source code) and solutions to be evaluated on the test set
using automated metrics (which we will run on our servers). The
current leaderboards will be visible to everyone.
  - Stage 2 will start on **September 10, 2020**. On September 10,
2020 the source code submission system will be locked, and the best
performing systems will be evaluated over the next month using crowd
workers.

******************************************************************************************************
                                                  CHALLENGE DESIGN
******************************************************************************************************
Detailed description: http://convai.io/ConvAI3_ClariQ2020.pdf

In Stage 1, we provide to the participants the datasets that include:

  - User Request: an initial user request in the conversational form,
e.g., “What is Fickle Creek Farm?”, with a label reflects if
clarification is needed to be ranged from 1 to 4;
  - Clarification questions: a set of possible clarifying questions,
e.g., “Do you want to know the location of fickle creek farm?”;
  - User Answers: each question is supplied with a user answer, e.g.,
“No, I want to find out where can I purchase fickle creek farm
products.”

To answer RQ1: Given a user request, return a score [1−4] indicating
the necessity of asking clarifying questions.

To answer RQ2: Given a user request which needs clarification, returns
the most suitable clarifying question. Here participants are able to
choose: (1) either select the clarifying question from the provided
question bank (all clarifying questions we collected), aiming to
maximize the precision, (2) or choose not to ask any question (by
choosing Q0001 from the question bank.)

The dataset is stored in the following repository
https://github.com/aliannejadi/ClariQ, together with evaluation
scripts and baseline.

## Stage 2: human-in-the-loop

The TOP-5 systems from Stage 1 are exposed to real users. Their
responses—answers and clarifying questions—are rated by the users. At
that stage, the participating systems are put in front of human users.
The systems are rated on their overall performance. At each dialog
step, a system should give either a factual answer to the user’s query
or ask for clarification. Therefore, the participants would need to:

- ensure their system can answer simple user questions
make their own decisions on when clarification might be appropriate
- provide clarification question whenever appropriate
interpret user’s reply to the clarifying question

The participants would need to strike a balance between asking too
many questions and providing irrelevant answers. Note that the setup
of this stage is quite different from Stage 1. Participating systems
would likely need to operate as a generative model, rather than a
retrieval model. One option would be to cast the problem as generative
from the beginning and solve the retrieval part of Stage 1, e.g., by
ranking the offered candidates by their likelihood. Alternatively, one
may solve Stage 2 by retrieving a list of candidate answers (e.g., by
invoking Wikipedia API or the Chat Noir API that we describe in the
document) and ranking them as in Stage 1.


******************************************************************************************************
                                                  WORKSHOP
******************************************************************************************************
The best systems will have a chance to present their work at the 5th
international workshop on Search-Oriented Conversational AI (SCAI) at
EMNLP in November. Apart from the challenge, the workshop features
invited speakers and peer-reviewed submissions. The submission system
is currently open (until August 15), the details can be found under
https://scai.info/2020/

*****************************************************************************************************
                                                   ORGANIZERS
******************************************************************************************************
 * Mohammad Aliannejadi (University of Amsterdam)
 * Julia Kiseleva (Microsoft Research & AI)
 * Jeff Dalton (University of Glasgow)
 * Aleksandr Chuklin (Google Research)
 * Mikhail Burtsev (MIPT)

    ---------------------------------------------------------------
    For news of CHI books, courses & software, join CHI-RESOURCES
     mailto: [log in to unmask]

    To unsubscribe from CHI-ANNOUNCEMENTS send an email to
     mailto:[log in to unmask]

    For further details of CHI lists see http://listserv.acm.org
    ---------------------------------------------------------------

ATOM RSS1 RSS2