From Raw Interaction Traces to Actionable Indicators: Lessons Learned

Résumé En

ELIPSS (Étude Longitudinale par Internet Pour les Sciences Sociales) is a probability-based Internet panel dedicated to social sciences, inspired by the dutch LISS Panel. Each panel member is provided with a touch screen tablet, through which academic surveys are delivered monthly. The study is in its 6th year of existence and has hit its stride with around 2500 panel members remaining and 69 surveys conducted so far.During the pilot study, the ELIPSS team designed specific management processes and interaction strategies to maximize participation and minimize attrition; in support of which custom software tools were developed, ranging from an Android questionnaire delivery application to web services facilitating fieldwork management. This growing ELIPSS toolbox was extended in 2017 with a real time analytics dashboard.Sustaining a daily flow of multi-channel interactions with panel members is a focus point of the panel management team’s efforts. Phone calls, electronic and traditional mail, SMS, and lastly in-app notification messages were incrementally adopted and combined. Both individual and batch delivery are supported where relevant. Detailed traces of all these activities (connection times, communication histories, answering tracks, assistance requests…) were collected along the way, yielding a wealth of behavioral metadata. However, tapping into this potential to provide useful metrics and indicators remains a challenge for several reasons. First, the adaptive rather than rigidly foreplanned design process of tools and workflows occasioned heterogeneities in data shape and contents through time, that must be reconciled. Second, “quick-fix” solutions were sometimes unavoidable to pragmatically meet the needs of continuously ongoing fieldwork, and introduced numerous idiosyncrasies in the data model.This presentation will focus on the lessons learned from our best efforts in the experimental curation, documentation and aggregation of these raw interaction-traces to produce a usable paradata set capable of providing useful indicators to inform real-time decisions.