ACM SIGCHI General Interest Announcements (Mailing List)


Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Condense Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
text/plain; charset="iso-8859-1"
Mon, 11 Apr 2016 09:18:04 +0000
Judith Redi - EWI <[log in to unmask]>
"ACM SIGCHI General Interest Announcements (Mailing List)" <[log in to unmask]>
Judith Redi - EWI <[log in to unmask]>
text/plain (112 lines)

                                                     Call for Papers


                                    PQS 2016 special session "Crowdsourcing"

Link to the special session:

           Part of the fifth ISCA/DEGA Workshop on Perceptual Quality of Systems

                                                        PQS 2016

                                                   29-31 August 2016

                                                     Berlin, Germany



Focus and objectives of the special session:

Crowdsourcing has become a valuable tool for studying and evaluating the perceptual quality of systems. It provides an easy, fast, and cost effective way to access a large number of diverse users. However, due to the highly uncontrolled environment in which crowd-based experiments are conducted, well-established lab test setups and experimental methodologies cannot be deployed without adaptions. These adaptions might involve technical changes of the test setup to support the remote devices of the participants, or methodical changes like the introduction of reliability checks to verify that the experimental task has been understood and executed properly.

In this session, we aim to foster the discussion between researchers in the field of crowdsourcing and perceptual studies to further close the gap between subjective testing and crowdsourcing. We solicit contributions addressing novel techniques to ease the use of crowdsourcing for the study of the perceptual quality of systems, successful and unsuccessful examples of crowd-sourced subjective studies that can help deriving general best practices or pitfalls, and methodical approaches that, e.g.,  enable better reproducibility and comparability of crowdsourcing studies

Special Session Organizers:

- Judith Redi, TU Delft, The Netherlands (j.a.redi /at/

- Matthias Hirth, University of Würzburg, Germany (matthias.hirth /at/

- Tim Polzehl, TU Berlin, Germany (tim.polzehl /at/

Submissions are solicited on all aspects of Crowdsourcing including (but not limited to):

- Crowdsourcing as methodology for studying and predicting the perceptual quality of systems

- Test design including adaptation of classic psychometric and user testing methodologies to a crowdsourcing environment

- Robust data analysis methodologies, quality assurance and unreliability detection, reproducibility of results

- Incentive design and deployment

- Tools and platforms providing enhanced support for crowd-based perceptual quality testing

- Inherent biases, limitations and trade-offs of crowd-centered approaches to the evaluation of perceptual quality of systems

- Pitfalls leading to the failure of previous studies and lessons learned from successful experiments

- Crowdsourcing usage in user behavior analysis and modelling

- Collaborative crowdsourcing and multi-party setup

- Crowd workers, their motivation and representativeness

- Ethical-, privacy or security-related issues with crowdsourcing in user-studies

Important Dates:

* Submission (extended abstracts, 1.5-2 pages): April 20, 2016

* Notification of acceptance: May 25, 2016

* Registration deadline for authors: June 5, 2016

* Camera-ready submission (full papers, 4+1 pages): June 20, 2016

Note: Papers submitted to the special session will undergo the same reviewing process by anonymous and independent reviewers as regular papers.

General information about PQS 2016:

The fifth ISCA/DEGA Workshop on Perceptual Quality of Systems aims to bring together leading professionals and scientists from different fields that are all related to perceptual quality, including Quality of Experience (QoE) and User Experience (UX). Supported by the International Speech Communication Association (ISCA), the Deutsche Gesellschaft für Akustik e.V. (DEGA) this event will serve as a forum facilitating for an interdisciplinary exchange of ideas between both academic and industrial researchers working on different aspects of perceptual quality of systems. Inaugurated in Herne/Mont Cenis (2003) and continued in Berlin (2006), Bautzen (2010) and Vienna (2013), PQS 2016 will return to Berlin, Germany, with this edition.

Dr. Judith Alice Redi
Assistant Professor
Department Intelligent Systems - Multimedia Computing
Electrical Engineering, Mathematics and Computer Science
Delft University of Technology
Mekelweg 4, HB 11.050
2628 CD Delft, The Netherlands
Tel: +31 15 2788543<tel:%2B31%2015%202788543>

    For news of CHI books, courses & software, join CHI-RESOURCES
     mailto: [log in to unmask]

    To unsubscribe from CHI-ANNOUNCEMENTS send an email to
     mailto:[log in to unmask]

    For further details of CHI lists see