What

Controlled experiments with participants can happen in-person or remotely, and the incentives for participation varies hugely, from no reward to monetary incentive, to vouchers, to school credit. When analysing the results of data collected in mixed methods, the incentives are not often considered in terms of the effects they can have on the data collected.

Why

After the pandemic, participant recruitment shifted heavily towards remote methods. Whilst usability labs still exist, remote experimentation is still common. Methods of remunerating participants are also seeing innovation. I became aware that I was collecting data using different incentive schemes and had not inspected it to see if these schemes were affecting the data. There was no common knowledge to suggest this would be the case and had not been researched or documented.

Action

I collated previously completed research that collected similar metrics from users (typing speed and similar measures) and re analysed the results using both environment (online/in-person) and remuneration (money/statistics about your performance/no reward). The results were surprising. Using the paid, in-person results as a benchmark I found that participants who took part online for no money, or for data alone, produced significantly higher quality data than online participants who were reimbursed with money.

Impact

This work highlighted that reimbursement method can have an effect upon the quality of data collected from user experiments, a phenomena previously undocumented and unconsidered. The results of this work highlight that intrinsic motivation in participants to take part (eg. learning about themselves, or simply being happy to contribute) can produce better data than simply paying participants. The ethical implications of this outcome are complex - do you risk a bias in your sample if you don’t provide a monetary reward?