How to Destroy Your Mturk Reputation AND Your Business Reputation at the Same Time- Lighting Buff

Don't let this happen to you!

On Amazon Mechanical Turk, reputation is everything. The reputation that a requester has with the workforce will directly translate  into the quality of workers who work on your task. If you have a stellar reputation, you will have the best workers and get your work done quickly; if you have a poor reputation, you will only get new workers with questionable results. Do not ruin your reputation before you have a chance to build one.

Aug 15, 2012, a day that requester Lighting Buff will remember for a long time to come. This was the day they published their first two hits. They published two batches, one batch of 2000 hits for $0.04 each and another batch of hits 30,000 for $0.02 each.

On the second batch of 30,000 hits, the requester rejected at least 40% of the work that was submitted.

So what was wrong with this, and why does it matter? Let's go step by step and list the problems and solutions to what was done wrong.

1. The second batch of hits was coded incorrectly and did not have good instructions. The requester wanted workers to check two boxes for the room that a lighting fixture would look best in. The requester used radio buttons instead of check boxes. Radio buttons only allow one option to be checked and check boxes allow multiple boxes to be checked. So right from the beginning, there was confusion with what needed to be done for payment.  
How to avoid this problem - The Hits should have been tested first in the sandbox to ensure they were coded correctly. The requester should have asked a coworker to look at the Hits and try to work on one before publishing them to the general workforce. The requester should have asked others if the instructions are clear. Also, these hits were relying upon a subjective question that some people may answer differently. When publishing hits of a subjective nature, requesters should NEVER reject. Only when the Hits have been proofread and verified should they have been published.

2. Almost a week later the requester started to approve and reject Hits.  During this time, workers are all communicating and wondering if they took a big risk on a new requester. They are getting restless. Amazon allows requesters 30 days to reject/approve hits, but requesters should not wait this long, especially a new requester.
How to avoid this problem - Publish a small test batch of 500 and approve/reject immediately. Do not publish 30,000 hits and wait over a week to approve them. By the time the rejections started rolling in, workers were ready to come down full force on this requester. Postings all over the internet flooded sites that rate requesters with negative reviews. Workers were angry at rejections over a subjective hit that was not coded properly. Within two days, Turkopticon had 20 reviews with the worst rating possible. The Hall of Fame/Shame on TurkerNation had a post dedicated to Lighting Buff with all negative reviews.

3. The requester speaks out and tries to get a grip on what happened and digs a deeper hole. The requester came to TurkerNation after seeing the reviews and tried to communicate because they had more Hits to publish. In doing so, they explained that they did not review each individual hit, but reviewed a few from each worker and rejected based on a percentage.
How to avoid this problem - Specifics were not revealed, but telling workers that their work was rejected and no one ever looked at it is a BAD IDEA. I am not saying that requesters should kiss workers asses, but try to look at things from their point of view. Workers are allowed to work on specific Hits based upon their approval percentage. Workers approval percentages are a top priority because many requesters require a very high approval percentage to work on the best paying jobs. When they are rejected for working on a hit that paid $0.02 each for a new requester they took a chance on out of good faith, the requester should be on their very best behavior and try to rebuild their reputation as quickly as possible.

4. Upon coming to TurkerNation, Lighting Buff informed workers they would like to reverse the rejections but do not know how. The community showed the requester how to reverse the rejections and tried to work with them on the reputation issue. The requester then informed the community they were publishing another batch of Hits and asked the people trying to help them to work on these new Hits. They then rejected some of the new Hits the TurkerNation community worked on.
How to avoid this problem - STOP! Fix your problem first. Reverse the rejections and try to rebuild your reputation before publishing any more hits. Then communicate with workers when you publish more hits and do not reject. If a worker makes an error, communicate with the community before you start rejecting.

Hopefully this has been helpful and informative. Unfortunately, Amazon does not make it easy for new requesters to get involved with mturk. This is why solution providers charge an arm and a leg to publish Hits. You don't have to pay those solution providers if you start by communicating with workers on TurkerNation before you publish your hits.