CHI-ANNOUNCEMENTS Archives

ACM SIGCHI General Interest Announcements (Mailing List)

CHI-ANNOUNCEMENTS@LISTSERV.ACM.ORG

Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
"M. Larson" <[log in to unmask]>
Reply To:
M. Larson
Date:
Wed, 18 Nov 2015 19:51:54 -0600
Content-Type:
text/plain
Parts/Attachments:
text/plain (93 lines)
==================================================================
NewsREEL Real-time Stream Recommendation Challenge at CLEF
http://www.clef-newsreel.org
==================================================================

Important Dates:

CLEF registration opens: 30 October 2015
Complete CLEF registration at the latest: 2 April 2016
Paper Submission Deadline: 25 May 2016
Feedback on Submission Status: 17 June 2016
Camera-ready Version Due: 1 July 2016
CLEF Conference and Labs (Evora, Portugal): 5-8 September 2016

We invite you to develop stream recommendation solutions to address a 
news recommendation task in two exciting settings. First, in Task 1, 
your solution has the chance to serve real users in a living lab. 
Second, in Task 2, you can “re-play” logged interactions to compare 
different algorithms on identical data. Task 2 can support you in 
developing your solution before deploying it online in Task 1. Continue 
reading for more details.

---- Task 1: Online Evaluation in a Living Lab ---

Task 1 is a living lab that provides task participants with access to 
A/B testing facilities of a commercial service. In this way, the task 
allows participants to evaluate the response of a real-world user 
population to different recommender system algorithms.

As soon as you sign up to participate, you can implement a system, and 
your system performance will appear on the public leaderboard. This will 
allow you to compare your performance as soon as possible to a baseline 
recommender.

The winner for this task will be determined on the basis of performance 
during the four week Challenge Window in April 2016. The official 
evaluation metric is the click-through-rate (CTR). CTR measures 
clicks/requests for all those requests to which a participating system 
has responded successfully within the time limit.

We encourage you to start developing and optimizing your algorithms as 
soon as possible. To support you in this respect, we suggest that you 
participate in Task 2 and also take advantage of the three Development 
Windows, during which we will record performance in order to give you a 
picture on where you stand.

The time frames are:

6-12 February 2016: Development Window 1
20-26 February 2016: Development Window 2
5-11 March 2016: Development Window 3
2-29 April 2016: Challenge Window

The Challenge Window is timed so that you can submit your paper to the 
CLEF Lab by the 25 May 2016 deadline. Note that you sign up for CLEF 
after 2 April, you will be allowed to submit a paper on your solution to 
the CLEF Lab, but you cannot win the competition. For this reason, we 
advise you to sign up as soon as possible, and make sure that your 
system is running stably by 2 April. If you need a virtual machine to 
run you solution, we will provide you with one.

--- Task 2: Offline Evaluation using Simulation Framework---

Task 2 allows you to run a comprehensive evaluation offline, comparable 
to the online evaluation of Task 1. The official evaluation metric is 
the click-through-rate (CTR), but we encourage participants to consider 
other metrics as well. We provide you with a data set and a framework 
that allows you to compare a variety of algorithms on identical data and 
reliably conclude which one performs best. Additionally, in Task 2, you 
gain insights on the technical challenges of running recommendation 
algorithms including complexity, scalability, and responsiveness. 
Besides prediction accuracy, you can run extensive experiments to 
determine which algorithm handles the most simultaneous load, which 
algorithm requires the least memory, or which algorithm can be most 
easily distributed. These factors reflect important constraints in 
industrial recommender systems.

You can find more information and also the CLEF registration link at:
http://www.clef-newsreel.org/how-to-participate

With questions or for more information, please contact:
[log in to unmask]

    ---------------------------------------------------------------
    For news of CHI books, courses & software, join CHI-RESOURCES
     mailto: [log in to unmask]

    To unsubscribe from CHI-ANNOUNCEMENTS send an email to
     mailto:[log in to unmask]

    For further details of CHI lists see http://listserv.acm.org
    ---------------------------------------------------------------

ATOM RSS1 RSS2