==================================================================
NewsREEL Real-time Stream Recommendation Challenge at CLEF
http://www.clef-newsreel.org
==================================================================
Important Dates:
CLEF registration opens: 30 October 2015
Complete CLEF registration at the latest: 2 April 2016
Paper Submission Deadline: 25 May 2016
Feedback on Submission Status: 17 June 2016
Camera-ready Version Due: 1 July 2016
CLEF Conference and Labs (Evora, Portugal): 5-8 September 2016
We invite you to develop stream recommendation solutions to address a
news recommendation task in two exciting settings. First, in Task 1,
your solution has the chance to serve real users in a living lab.
Second, in Task 2, you can “re-play” logged interactions to compare
different algorithms on identical data. Task 2 can support you in
developing your solution before deploying it online in Task 1. Continue
reading for more details.
---- Task 1: Online Evaluation in a Living Lab ---
Task 1 is a living lab that provides task participants with access to
A/B testing facilities of a commercial service. In this way, the task
allows participants to evaluate the response of a real-world user
population to different recommender system algorithms.
As soon as you sign up to participate, you can implement a system, and
your system performance will appear on the public leaderboard. This will
allow you to compare your performance as soon as possible to a baseline
recommender.
The winner for this task will be determined on the basis of performance
during the four week Challenge Window in April 2016. The official
evaluation metric is the click-through-rate (CTR). CTR measures
clicks/requests for all those requests to which a participating system
has responded successfully within the time limit.
We encourage you to start developing and optimizing your algorithms as
soon as possible. To support you in this respect, we suggest that you
participate in Task 2 and also take advantage of the three Development
Windows, during which we will record performance in order to give you a
picture on where you stand.
The time frames are:
6-12 February 2016: Development Window 1
20-26 February 2016: Development Window 2
5-11 March 2016: Development Window 3
2-29 April 2016: Challenge Window
The Challenge Window is timed so that you can submit your paper to the
CLEF Lab by the 25 May 2016 deadline. Note that you sign up for CLEF
after 2 April, you will be allowed to submit a paper on your solution to
the CLEF Lab, but you cannot win the competition. For this reason, we
advise you to sign up as soon as possible, and make sure that your
system is running stably by 2 April. If you need a virtual machine to
run you solution, we will provide you with one.
--- Task 2: Offline Evaluation using Simulation Framework---
Task 2 allows you to run a comprehensive evaluation offline, comparable
to the online evaluation of Task 1. The official evaluation metric is
the click-through-rate (CTR), but we encourage participants to consider
other metrics as well. We provide you with a data set and a framework
that allows you to compare a variety of algorithms on identical data and
reliably conclude which one performs best. Additionally, in Task 2, you
gain insights on the technical challenges of running recommendation
algorithms including complexity, scalability, and responsiveness.
Besides prediction accuracy, you can run extensive experiments to
determine which algorithm handles the most simultaneous load, which
algorithm requires the least memory, or which algorithm can be most
easily distributed. These factors reflect important constraints in
industrial recommender systems.
You can find more information and also the CLEF registration link at:
http://www.clef-newsreel.org/how-to-participate
With questions or for more information, please contact:
[log in to unmask]
---------------------------------------------------------------
For news of CHI books, courses & software, join CHI-RESOURCES
mailto: [log in to unmask]
To unsubscribe from CHI-ANNOUNCEMENTS send an email to
mailto:[log in to unmask]
For further details of CHI lists see http://listserv.acm.org
---------------------------------------------------------------
|