Academic Survey Publishing

Successfully publishing an academic survey on Mturk can be an exhilarating experience. Watching the participants submit and monitoring the progress of your study is truly exciting. When it comes time to examine the data though, some researchers are left disenchanted with the platform because of poor quality responses or  comments which reveal there is a lot to learn on Mturk. 

Three keys to being a quality requester on Mturk 

1. Your reputation matters on Mturk. On the participant side, Amazon has built in an approval percentage. It is a generic pop-up on the worker side which lets participants know if you are going to reject their work or approve it. Many independent worker review sites are also linked to Mturk to allow participants a forum to report requesters who are behaving in an unethical manner. Only reject a participant's submission if it is outright theft. If it is just poor quality data, add them to a disqualifier qualification and do not allow them to participate in any future studies. 

2. Use qualifications wisely or even better, screen participants prior to allowing access. You could use Amazon's premium qualifications, but these are just a way for Amazon to reduce your budget. Pay a small amount and screen participants yourself. Have them self report their demographics and when your initial small screening study is completed, send them an invitation to your main study via a separate HIT. 

We have found that prescreening reduces unusable responses to less than 1%. If using qualifications alone, always set to a minimum of 98-99% approval or greater. Mturk is a workplace, not a classroom. The participants who do not read instructions, answer randomly and are inattentive are the ones you will find under the 98% approval percentage. 

3. Always be civil in any email correspondence. We know that survey design takes long hours of hard work and it is easy to take critical emails personally. You may receive emails from participants that seem to attack you personally or your work. Sometimes no response is the best response but if a response is required, always be professional and do your best to respond in a professional manner. 

As always if you do not want to deal with the complexities of self publishing on mturk, contact us at design@mturkdata.com