Thursday, August 30, 2012

How to Destroy Your Mturk Reputation AND Your Business Reputation at the Same Time- Lighting Buff

Don't let this happen to you!

On Amazon Mechanical Turk, reputation is everything. The reputation that a requester has with the workforce will directly translate  into the quality of workers who work on your task. If you have a stellar reputation, you will have the best workers and get your work done quickly; if you have a poor reputation, you will only get new workers with questionable results. Do not ruin your reputation before you have a chance to build one.

Aug 15, 2012, a day that requester Lighting Buff will remember for a long time to come. This was the day they published their first two hits. They published two batches, one batch of 2000 hits for $0.04 each and another batch of hits 30,000 for $0.02 each.

On the second batch of 30,000 hits, the requester rejected at least 40% of the work that was submitted.

So what was wrong with this, and why does it matter? Let's go step by step and list the problems and solutions to what was done wrong.

1. The second batch of hits was coded incorrectly and did not have good instructions. The requester wanted workers to check two boxes for the room that a lighting fixture would look best in. The requester used radio buttons instead of check boxes. Radio buttons only allow one option to be checked and check boxes allow multiple boxes to be checked. So right from the beginning, there was confusion with what needed to be done for payment.  
How to avoid this problem - The Hits should have been tested first in the sandbox to ensure they were coded correctly. The requester should have asked a coworker to look at the Hits and try to work on one before publishing them to the general workforce. The requester should have asked others if the instructions are clear. Also, these hits were relying upon a subjective question that some people may answer differently. When publishing hits of a subjective nature, requesters should NEVER reject. Only when the Hits have been proofread and verified should they have been published.

2. Almost a week later the requester started to approve and reject Hits.  During this time, workers are all communicating and wondering if they took a big risk on a new requester. They are getting restless. Amazon allows requesters 30 days to reject/approve hits, but requesters should not wait this long, especially a new requester.
How to avoid this problem - Publish a small test batch of 500 and approve/reject immediately. Do not publish 30,000 hits and wait over a week to approve them. By the time the rejections started rolling in, workers were ready to come down full force on this requester. Postings all over the internet flooded sites that rate requesters with negative reviews. Workers were angry at rejections over a subjective hit that was not coded properly. Within two days, Turkopticon had 20 reviews with the worst rating possible. The Hall of Fame/Shame on TurkerNation had a post dedicated to Lighting Buff with all negative reviews.

3. The requester speaks out and tries to get a grip on what happened and digs a deeper hole. The requester came to TurkerNation after seeing the reviews and tried to communicate because they had more Hits to publish. In doing so, they explained that they did not review each individual hit, but reviewed a few from each worker and rejected based on a percentage.
How to avoid this problem - Specifics were not revealed, but telling workers that their work was rejected and no one ever looked at it is a BAD IDEA. I am not saying that requesters should kiss workers asses, but try to look at things from their point of view. Workers are allowed to work on specific Hits based upon their approval percentage. Workers approval percentages are a top priority because many requesters require a very high approval percentage to work on the best paying jobs. When they are rejected for working on a hit that paid $0.02 each for a new requester they took a chance on out of good faith, the requester should be on their very best behavior and try to rebuild their reputation as quickly as possible.

4. Upon coming to TurkerNation, Lighting Buff informed workers they would like to reverse the rejections but do not know how. The community showed the requester how to reverse the rejections and tried to work with them on the reputation issue. The requester then informed the community they were publishing another batch of Hits and asked the people trying to help them to work on these new Hits. They then rejected some of the new Hits the TurkerNation community worked on.
How to avoid this problem - STOP! Fix your problem first. Reverse the rejections and try to rebuild your reputation before publishing any more hits. Then communicate with workers when you publish more hits and do not reject. If a worker makes an error, communicate with the community before you start rejecting.

Hopefully this has been helpful and informative. Unfortunately, Amazon does not make it easy for new requesters to get involved with mturk. This is why solution providers charge an arm and a leg to publish Hits. You don't have to pay those solution providers if you start by communicating with workers on TurkerNation before you publish your hits.



8 comments:

  1. Yikes! I'm glad I missed out on this particular requester. People like them ruin it for new requesters everywhere...

    ReplyDelete
  2. Hi, I work for the company you are writing about. We did make some mistakes, we were beginning to learn how to use this system. The person who was in charge of setting this up initially was only in the office once a week, hence the once a week actions. Unfortunately, Amazon was not providing any training like they used to offer in the past. We corrected our mistakes and tried to make good with the people that we harmed. We have since hired someone who has experience to work with Amazon Turk and have had nothing but good things come from it on our end and the workers' end.

    ReplyDelete
    Replies
    1. Yes this may be beating a dead horse but this comment:

      " We corrected our mistakes and tried to make good with the people that we harmed. "

      is not completely true. You did not try to make good with all of the people that you harmed. You only tired to make good with the people that were on certain forums that you had harmed.

      Again, this is a dead horse and I am sure this comment will get deleted too but I still felt the need to post it.

      Delete
    2. Yes, you have made some significant changes. I am sure that the workers appreciate it. It is unfortunate that the reputation received still haunts you. Hopefully this post will help other Amazon Mechanical Turk requesters avoid the problems that you faced.

      With the high turnover on Mturk, 6 months is a long time and probably very few workers remember this incident. The long time and most experienced Turkers do remember and your reviews on other websites like Turkopticon are going to remain published for a long time.
      That is why it is so important to be careful when you are a requester and starting out on Mturk. That is also the reasoning behind this blog so that hopefully this helps others.

      I am also aware that you have hired someone with valuable mturk experience to help you and that you have gotten some great workers to help you move forward. Hope you are getting the results you need with the changes.

      Delete
  3. I really appreciate your post and you explain each and every point very well. Thanks for sharing this information. And I’ll love to read your next post too.This will help me to complete my task , tips are really helpful for me.

    corporate reputation

    ReplyDelete
  4. Here's the TO for them. It explains the problem with a bit more detail.
    http://turkopticon.differenceengines.com/main/search

    ReplyDelete
  5. Margret, your link is just to the search, this is the requester review link - http://turkopticon.differenceengines.com/ASTSL00H0ODD1 As you can see the requester has gotten some positive reviews after the initial problems, but the old reviews are still there. That is really the intention of this post so that other requesters do not suffer the same fate. This requester is working with only qualified workers now so these old reviews are really irrellavant. I have this qualification and I am more than happy to work for them. It is unfortunate what happened, but there is nothing that can be done to repair that reputation now because of the nature of TO and worker forums.

    ReplyDelete
  6. Count me in as one of the suckers affected by Lighting Buff's errors. I had a big batch of 200+ HITs rejected by Lighting Buff, and it haunts me to this day. I don't work a ton of HITs on MTurk, but have been trying to get back at it the past few months. I did tons of cheap 2 penny business card transcription HITs to help raise my rating. I've finally raised it to over 90%. But before this incident, it was close to 98%, and it will take me 8 more months to raise it to just 95%! I contacted them requesting something be done at that time, and have contacted Amazon twice. Nada.

    ReplyDelete