I just finished listening to Planet Money’s #600 episode titled “The People Inside Your Machine”. Here’s a link to the podcast episode. The show briefly discusses the origin of the MTurk program (something I didn’t know!). Planet Money is an economics podcast and so the hosts are particularly interested in the amount of money that Turkers are earning whist completing jobs. It turns out the answer is “not that much”. The show begins to question the ethics of this, highlighting how difficult it is to earn a reliable wage from working on MTurk. But the focus then quickly turns to whether or not Turkers are helping to train computers to take the jobs of humans essentially.
One particular aspect of the episode that I found interesting was the investigation into the Turkers themselves; aiming to understand who they are. In this show, the presenters hide a “secret message” as they call it into the HIT they place on MTurk. This message asks the Turker to contact Planet Money and have a chat. Through these chats the presenters find out that most of their Turkers are in the US (not surprising seeing as Mechanical Turk requires you to be based in the US, anyone outside is doing something clever). One other trait that the presenters on the episdoe highlight is that many of their Turkers has problems with social anxiety, making Turk work far preferable to them than work outside the home. As one Turker in the episode puts it: “It’s enough to say that my temperament and the job market don’t see eye to eye.”
In HCI research, particularly at CHI recently there have been many studies run using Turkers as participants, rather than using the standard bring-people-into-the-lab method of recruitment. I’m a big fan of this approach, I personally have begun to use Citizen Science style experiments in my research, asking people to complete experiments and learn a bit about themselves as a reward. The HCI community is currently dealing with issues of ethics; the HIT system on Mechanical Turk currently doesn’t come close to matching the rate that participants normally receive. Just because that’s standard on the site, does that make it right?
One big perceived positive of recruiting through MTurk is the diversity of potential participants. Whereas many HCI, and indeed Psychology studies suffer from recruiting only WEIRD (Western, Educated, Industrialized, Rich, and Democratic) participants, MTurk represents a broader sample of the population (though still requiring certain things like access to a computer and an internet connection). There have been studies aimed at understanding the demographics of Turkers, showing that your average Turker is likely to be younger than 34 earning less than $10k a year.
It’s important to know this information about the Turkers. When running studies with the aim to collect performance data about humans in particular tasks, you need to know who you’re testing on, and that they are representative of a larger population. But is it enough to just know this demographic information? The discussion in this podcast got me thinking.
How does a participant’s social anxiety affect their performance in a task? For many studies within HCI this won’t likely be a problem: there isn’t a clear reason why clicking targets on a screen would be affected by such anxiety. However, other studies certainly would be. For instance, studies asking about facebook behaviour, or trust. Although definitely not a rigourous analysis or in depth study into Turkers, this podcast highlights the possibility that Turkers could be more likely to be socially anxious than average. Meaning any results from studies into areas that would be affected by such a condition would be seriously compromised, and unrepresentative of the rest of the population. The conclusions drawn might suggest that people were more fearful or angry about certain situations than is necessarily true.
These studies, when run with the usual participants recruitment methods, would have no reason to suspect their participants were not representative of average social anxiety levels. However here we have some evidence that such an assumption cannot be made when using Turkers as participants.
It’s not a ground breaking conclusion to come to: that more information needs to be known about Turkers as they are used in HCI research. But this podcast has highlighted one key area that genuinely could be confounding the results of current experiments.