Tips for Academic Requesters on Mturk

Quick post for academic survey requesters on Amazon Mechanical Turk.

The comment below was made by a survey requester 

 



It is not the worker's fault that you are not getting quality results. There are ways to get excellent results with your surveys on Mturk if you follow a few simple rules.

1. Set your "HITs approved" qualification to ABOVE 97%, NOT 95%. Mturk is work, not school. A, B, C, D, F does not apply here.
A workers are at approval rating of 99% and 98%.
B workers are at approval rating of 97% and 96%.
F workers are anything below that.
The vast majority of good turkers are above the 97 percent approval rating.  It is very hard for any scamming worker to maintain an average above this percentage. There are two exceptions to this rule. Some new workers will work for the wrong requester and have their approval rating damaged with unjust rejections. Because new workers have fewer hits submitted, rejections influence their approval percentage more. The other exception is workers who are not in the United States. These workers are very limited by the hits they can accept and are forced to work for more requesters who use "Plurality" to grade hits. There is no way to avoid rejections when requesters use the ridiculous grading system of majority rules.

 2. Set your "HITs submitted" qualification to over 5000. This goes hand in hand with the above comment. It is fairly easy to maintain a high approval percentage with fewer HITs submitted. Personally, if I was setting this qualification, I would use 10,000 HITs submitted, because it is almost impossible for a scamming worker to achieve over 10,000 hits submitted with over a 97 percent approval rate.

3. Use attention check questions in your surveys. If a worker is not reading the instructions for each answer, they will choose an incorrect answer. An example would be.....
"How are you feeling right now? Although we would like to know how you are feeling, please select "B" so we know you are paying attention."

4. Pay fairly. Academic requesters are notorious on Mturk for being some of the worst paying requesters.  The ABSOLUTE MINIMUM pay you offer should be $0.10 per minute. This works out to $6.00 per hour. Although this is below the federal minimum wage, good turkers will accept surveys for this pay. The best academic requesters pay between $0.20-$0.30 per minute.
Mturk is not charity and it is not volunteer work. If you do not have the budget to work within these guidelines, you need to seek additional funding or do not publish your hits. It is highly unethical to publish hits that pay below minimum wage. If you search Mturk with the word "survey" you will see results from many unethical academic requesters who have no respect for the workers who complete their tasks. Also, because you see these tasks, it means they are not getting completed and are just sitting.

5. Treat your workers with respect and dignity. Workers are not numbers and statistics. Workers are not lab rats. Workers are people and should be treated with respect.

6. DO NOT BLOCK WORKERS! This is the standard response Amazon gives to academic requesters who require unique results for each questionnaire submitted. Reference post here IsaacM@AWS  This is not the right way to ensure unique results, and had in the past been disastrous for workers. Make a qualification HIT for your survey. After the worker completes the survey, revoke the qualification. If blocks are not done correctly, Amazon will shut down a workers account for review.

As always, you can contact me on TurkerNation for help.