Conference and Labs of the Evaluation Forum
Bucharest, Romania, September 21-24, 2021
CLEF is a leading annual international conference exploring topics in
information access technologies. CLEF uniquely combines evaluation
laboratories and workshops with research presentations, panels, poster
and demo sessions. CLEF 2021 (http://clef2021.clef-initiative.eu/)
will be held on 21-24 September 2021 with University "Politehnica" of
Bucharest, in Bucharest, Romania.
Researchers and practitioners from all areas of information access and
related communities are invited to submit proposals for Evaluation
Labs for consideration for inclusion in CLEF 2021 lab programme. Lab
proposals will be reviewed by the lab selection committee.
The CLEF Initiative (http://www.clef-initiative.eu/) is a
self-organized body whose main mission is to promote research,
innovation, and development of information access systems with an
emphasis on multilingual information in different modalities -
including text and multimedia - with various levels of structure. CLEF
promotes research and development by providing an infrastructure for:
- independent evaluation of information access systems,
- investigation of the use of unstructured, semi-structured,
highly-structured, and semantically enriched data in information
- creation of reusable test collections for benchmarking,
- exploration of new evaluation methodologies and innovative ways of
using experimental data,
- discussion of results, comparison of approaches, exchange of ideas,
and transfer of knowledge.
CLEF 2021 will be the twenty second edition of the current CLEF
conference launched as a continuation of the annual CLEF evaluation
campaign series which ran with great success from 2000 to 2009
contributing to the systematic evaluation of information access
systems, primarily through experimentation on shared tasks.
#Scope of CLEF Labs
Proposals are accepted for two different types of labs (see also the
2020 Labs at https://clef2020.clef-initiative.eu/index.php?page=Pages/labs.html):
- Evaluation Labs that are a "campaign-style" evaluation for specific
information access problems (during the twelve month period preceding
the conference), similar in nature to the traditional CLEF campaign
"tracks". Topics covered by campaign-style labs can be inspired by any
information access-related domain or task.
- Labs that follow a more classical "workshop" pattern, exploring
issues of evaluation methodology, metrics, processes, etc. in
information access and closely related fields, such as natural
language processing, machine translation, and human-computer
If the lab organisers of the proposal are new to CLEF or other shared
task evaluation campaigns, we highly recommend that a lab workshop
first be organised to discuss the format, the problem space, and the
practicalities of the shared task. The CLEF 2021 programme will
reserve about half of the conference schedule for lab sessions. The
labs will present their overall results "overview presentations"
during the plenary scientific paper sessions to allow non-participants
to get a sense of where the research frontiers are moving. Organisers
of each lab are expected to organise the separate sessions for their
lab at the conference so that they contain ample time for general
discussion and engagement by all participants - not just those
presenting campaign results and papers. Lab organisers should plan
time in their sessions for activities such as panels, demos, poster
sessions etc. as appropriate.
CLEF is always interested in receiving and facilitating innovative lab
proposals. Potential task proposers who are unsure of the suitability
of their task proposal or its format for inclusion at CLEF are
encouraged to contact the CLEF 2021 Lab Organizing Committee Chairs to
discuss its suitability or design at an early stage.
Lab proposals must provide sufficient information to judge the
relevance, timeliness, scientific quality, benefits for the research
community, and the competence of the proposers to coordinate the lab.
Each lab proposal should identify one or more organizers as
responsible for ensuring the timely execution of the lab. Proposals
should be 3 to 4 pages long and should provide the following
- Title of the proposed lab.
- A brief description of the lab topic and goals, its relevance to
CLEF and the significance for the field.
- A brief but clear statement of usage scenarios or domain to which
the activity is intended to contribute, including the evaluation setup
- Details on the lab organizer(s), including identifying the task
chair(s) responsible for ensuring the running of the task. This should
include details of any previous involvement in organising or
participating in evaluation tasks at CLEF or similar campaigns.
- The planned format of the lab, i.e., campaign-style ("track") or workshop.
- Is the lab a continuation of an activity from previous year(s) or a
new activity? For activities continued from previous year(s):
Statistics from previous years (number of participants/runs for each
task), a clear statement on why another edition is needed, an explicit
listing of the changes proposed, and a discussion of lessons to be
learned or insights to be made. For new activities: A statement on why
a new evaluation campaign is needed and how the community would
benefit from the activity.
- Details of the expected target audience, i.e., who do you expect to
participate in the task(s), and how do you propose to reach them.
- Brief details of tasks to be carried out in the lab. The proposal
should clearly motivate the need for each of the proposed tasks and
provide evidence of its capability of attracting enough participation.
The dataset which will be adopted by the Lab needs to be described and
motivated in the perspective of the goals of the Labs; also
indications on how the dataset will be shared are useful. It is fine
for a lab to have a single task, but labs often contain multiple
closely related tasks, needing a strong motivation for more than 3
tasks, to avoid useless fragmentation.
- Expected length of the lab session at the conference: half-day, one
day, two days. This should include very high level details of the
planned structure of the session, e.g., participant presentations,
invited speaker(s), panels, etc., to justify the requested session
- Arrangements for the organization of the lab campaign: who will be
responsible for activities within the task; how will data be acquired
or created, what tools or methods will be used, e.g., how will
necessary queries be created or relevance assessment carried out; any
other information which is relevant to the conduct of your lab.
- If the lab proposes to set up a steering committee to oversee and
advise its activities, include names, addresses, and homepage links of
people you propose to be involved.
Lab proposals have to be submitted at the following address:
choosing the “CLEF 2021 Lab Proposals” track.
Each submitted proposal will be reviewed by the CLEF 2021 Lab
Organizing Committee. The decision will be sent by email to the
responsible organizer by September 15, 2020. The final length of the
lab session will be determined based on the overall organization of
the conference and the number of participant submissions received by a
#Advertising Labs at CLEF 2020 and ECIR 2021
Organizers of accepted labs are expected to advertise their labs at
both CLEF 2020 (22-25 September 2020, online event, Thessaloniki,
Greece) and ECIR 2021 (March 28 - April 1st 2021, Lucca, Italy). At
least one lab representative should participate in these events.
Advertising at CLEF 2020 will consist of running a break-out session
to discuss the lab with prospective participants, and
advertising/announcing it during the closing session.
Advertising at ECIR 2021 will consist of submitting a short lab
description to be included in the ECIR 2021 proceedings (submission by
15 October 2020), advertising the lab in a booster session during ECIR
2021, and presenting a poster about the lab.
#Mentorship Program for Lab Proposals from newcomers
As in the last year, CLEF 2021 will run a mentorship program to
support the preparation of lab proposals for newcomers to CLEF.
However due to the COVID-19 pandemic and the schedule shift, this year
will be a lighter and more accelerated process than in the previous
year. We encourage newcomers to refer to Friedberg et al. (2015) for
initial guidance on preparing their proposal: Friedberg I, Wass MN,
Mooney SD, Radivojac P. Ten simple rules for a community computational
challenge. PLoS Comput Biol. 2015 Apr 23;11(4):e1004150.
The CLEF newcomers mentoring program offers help, guidance, and
feedback on the writing of your draft lab proposal by assigning a
mentor to you, who help you in preparing and maturing the lab proposal
for submission. If your lab proposal falls into the scope of an
already existing CLEF lab, the mentor will help you to get in touch
with those lab organizers and team up forces.
Lab proposals for mentorship have to be submitted at the following address:
choosing the “CLEF 2021 Lab Mentorship” track.
Due to the COVID-19 pandemic, this year’s schedule for the lab
proposal submission has been significantly changed:
- Requests for accelerated mentorship submission (only newcomers): 24 July 2020
- Accelerated mentorship period starts: 27 July 2020
Lab proposals submission (both newcomers and veterans): 26 August 2020
- Notification of lab acceptance: 15 September 2020
- Advertising labs at CLEF 2020: 22-25 Sept. 2020, online event
- Submission of short lab description for ECIR 2021: 15 October 2020
- Lab registration opens: 5 November 2020
- Advertising labs at ECIR 2021: 28 March-1st April 2021, Lucca, Italy
#CLEF 2021 Lab Chairs
Alexis Joly, Inria, LIRMM, University of Montpellier, France
Maria Maistro, University of Copenhagen, Denmark
Florina Piroi, Vienna University of Technology, Austria
#CLEF 2021 Lab Accelerated Mentorship Chairs
Lorraine Goeuriot, Université Grenoble Alpes, France
Henning Müller, University of Applied Sciences Western Switzerland, Switzerland
On behalf of the General Chairs,
Prof. Bogdan IONESCU
ETTI - University Politehnica of Bucharest