Uncovering the dark side of the moon: Lessons learned from unexpected
outcomes of QoE experiments”

The deadline for this special session has been extended to 15 March 2015.

Instructions for authors submitting a paper to a special session:

After submitting your paper, please inform the organizers of the special session by email. Please also send an email to the special session co-chairs (katrien.demoor@item.ntnu.no / fliegek@fel.cvut.cz) containing the title of your paper and the session to which you have submitted a paper.

Motivation and Objectives:

Researching the QoE of multimedia systems is a challenging endeavour: not only have human perception and experience turned out to be highly complex subjects of study, but also underlying methods for quality assessment, modelling and validation – being far from perfect – have to keep evolving and improving. Consequently, researchers and practitioners in this domain are frequently confronted with unexpected outcomes of their experiments and studies. Prime examples are inconclusive results (caused e.g. by confounding factors) as well as outcomes that prevented the experimental hypothesis from being tested at all. In such cases,
experiments are typically refined and repeated, with only the strong/conclusive results being finally reported to the outside world.

However, the drawback of this situation is that common recurring problems and valuable lessons learned remain unreported ‘in the dark’. However, bringing them to light by sharing them would not only save time and effort of other QoE researchers about to conduct similar
experiments. It would also accelerate scientific progress in the field since such problems typically point towards important research challenges for the QoE community at large.

To address above issues, this session aims to provide a dedicated forum that enables and encourages QoE researchers and practitioners to share valuable lessons learned from past experiments so that common (and not so common!) pitfalls are avoided and methodologies improved, thereby effectively raising the QoE community’s scientific standards. Furthermore, on a higher level, this session shall help to explicitly formulate recurring or hard problems (e.g. expected user behaviour in crowdsourcing, no-reference metrics design) as research challenges for the QoE community and to jointly identify appropriate approaches and methods to address them.


Topics in the focus of this session (non-exhaustive):

– Lessons learned from quality assessment, modelling and validation research

– Unexpected outcomes and results from subjective quality experiments and the reasons behind

– Invalid algorithms and test methods that appeared valid at first glance

– Sources of errors and confounding factors and how to address them in scientific practice

– Seemingly confounding factors that turned out to be irrelevant (e.g. having age-balance within an experiment’s subject pool)

– Errors frequently observed in pre-existing papers and articles

– Recurring problems (e.g. unexpected user behaviour) in challenging experimental contexts (e.g. crowdsourcing, field trials)

– Results and lessons learned uncovering the nature of “hard” problems in quality research (e.g. no-reference metrics design)

– Evidence for commonly believed best practices (e.g. balancing quality levels in subjective experiments) vs. easy mistakes (e.g. not balancing quality levels)

– Improved test methodologies and experimental designs (based on lessons learned)
Short biography of the organizers

Raimund Schatz is Senior Researcher at the Telecommunications Research Center Vienna (FTW) and Manager of the User-centered Interaction and Communication Economics Department. He holds an Msc. in Telematics (TU-Graz), a PhD in Informatics (TU-Vienna), an MBA and an MSc. in International and Finance Management (both from Open University, UK). Dr. Schatz leads FTWs research projects (ACE, ACE 2.0, ACE 3) on Quality-of-Experience assessment and monitoring of broadband services in wireless and wireline networks, conducted together with a number of industry partners in the telecom industry. Furthermore, he is actively involved in various QoE-related EU projects and networking activities, including Optiband (FP7), CELTIC QuEEN and COST Action IC1003 (Qualinet) as well as the organization of various QoE-related workshops and events (e.g. Special Session Chair for QoMEX 2013; Organizer of PQS 2013, QoENAM 2014 and QoE-FI 2015). Being an active member of ACM and IEEE, he is author of more than 90 publications in the areas of Quality of Experience, HCI and Pervasive Computing.
Lucjan Janowski is Assistant Professor at AGH University of Science and Technology, Department of Telecommunication, Poland. He received his M.S. and PhD degrees from AGH in 2002 and 2004 respectively. Since 2005, he has performed research on QoE as well as network traffic analysis and modeling in a number of projects at various institutions, including LAAS-CNRS (Toulouse, FR), University of Geneva (Geneva, CH), Institute for Telecommunication Sciences (Boulder, Colorado, US), Carnegie Mellon University (Pittsburgh, US) and Telecommunications Research Center Vienna (FTW, AT). Dr. Janowski is actively involved in international fora such as Video Quality Experts Group (VQEG) where he currently serves as co-chair of the JEG-Hybrid project as well as Independent Laboratory Group (OLG) representative. Furthermore, he is leading the QoE journal task force in the COST Action IC1003 (Qualinet). His main research interests include subjective testing methods, QoE and traffic data analysis and modeling, video quality metrics and 3D video quality.
Margaret Pinson received her B.S. and M.S. degrees in computer science from the University of Colorado at Boulder, USA, in 1988 and 1990. Since 1988, she has been investigating improved methods for assessing video quality at the Institute for Telecommunication Sciences, an office of the National Telecommunication and Information Administration (NTIA/ITS), in Boulder, Colorado, USA. Mrs. Pinson is a Co-Chair of the Video Quality Experts Group (VQEG) and an Associate Rapporteur of Questions 2 and 12 in ITU-T Study Group 9. Her contributions and leadership within Study Group 9 played a key role in the approval of ITU-T Rec. P.913 (2014), Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. She encourages video quality research and development by administering the Consumer Digital Video Library (CDVL, www.cdvl.org<http://www.cdvl.org>, the largest database with high quality video which can be used for subjective experiments) and by making broadcast video sequences available on the CDVL.