It's a well known sentiment that the Masters qualification absolutely sucks for the majority of workers. Just to recap: Amazon pushes it on requesters, eliminating their chances at finding work, Amazon doesn't tell them how to get it, so even if they have one million HITs approved they might not be a “Master” and Amazon doesn't care at all about workers with any of these problems. Requesters can't even recommend workers as Masters, as a few have done for me. I have it on good authority that the Masters qualification is given to people who do random HITs on mTurk that Amazon puts up to test workers. I have no idea what these HITs pay, but if they pay little I doubt I've ever even noticed them. This really does explain why so many do not have Masters that deserve it. Take this how you will, a good journalist never names his sources. ;)
As a worker and requester (and consultant to requesters), though, I hate the Masters program doubly because as bad as it is for workers, it is even worse for requesters. Why?
- Requesters need to pay three times more to use Masters. Instead of paying 10¢ on every dollar, they pay 30¢.
- If a requester has a worker who they already know produces good work but that worker does not have Masters, that worker is barred from all of their HITs.
- Since there are a lot less Masters than Amazon says there are, work completed by Masters is done at a slower rate than that of work done by workers with sane custom qualification schemes or workers with system qualifications.
Well, points one and two are very well proven, but I get a lot of flack when I try to claim point three. Fortunately, Tim Edwards came to my aid.
Thanks to a lone spammer or several spammers, Mr. Edwards decided that he had had enough and that he was going to give Masters a shot. Amazon has been pushing it hard very recently, so it may have seemed like a wise thing to do. Unfortunately for Mr. Edwards, he might not have known a few things:
- He has several large datasets of workers. He could very easily have picked out hundreds of workers, the real Masters of his HITs, and qualified them with a custom qualification. I would have helped him out with this for as little as $30.
- There aren't as many Masters as Amazon shows on the worker site, as I explained in my previous post. This could change, but at the moment, there are around 5,000 of them. I have no idea how many of those are active accounts, or if that is even a real number.
Anyway, on to the fun part. Opportunities like this don't come around very often, and reeling from some very new workers being granted Masters while some of their most loyal fans are left wondering where they went wrong, I decided to take down the number remaining of Edward's HITs every fifteen minutes. Here are my results, with fun annotations!
Added 11/16: As ChapterFoe points out in the comments, these results, as shocking as they are, might not be conclusive. I've left this post alone for history's sake — but Masters are very unlikely to do do HITs that pay little (which I confirmed) so I may not have proved much by posting this at all. In any case, it is an interesting look at work done on Mechanical Turk. I still think work done by Masters will always be slower, but this level of slower might be pushing it and is not representative of the whole body of work done by Masters.
I'll post the full data once Edwards gets all his HITs back, but let me clue you in on some fun things right here and now:
- With 78000 HITs at 10¢, Edwards was spending 20% more than necessary, a whooping $1,560. For that price, I would have run his HIT ten times over.
- In the entire time Edwards was using Masters (twelve hours almost exactly), he got around 287 HITs done. If we figure each HIT takes one minute, using the crowd was actually 4.7x slower than if he had done all the work himself.
- For every hour of Masters HITs (he got around 24 HITs/hr with Masters), Edwards gets 1,610 by letting those with the mark of Cain (I kid!) work on his HITs. That's 67 times more work being done!
As to the Masters program? It's bad for workers and bad for requesters and needs some serious reforms. I think it's a good idea at heart, but Amazon's lack of transparency on the issue dooms it to blog posts like this one.
Until next time!
The problem with the Edwards hits was they did not pay enough. Master workers are not going to work in that pay scale.
ReplyDeleteBefore Edwards released his second batch with masters qualification, he had them available to everyone.
When this first batch of hits was released I was reading on mturkforum.com, workers were stating they were completing the hits at a rate of two per minute. That equates to $12 per hour. The workers were spending minimal time on the hits and submitting a lot of n/a after only spending 30 seconds searching. This is "almost" cheating and not what the requester was looking for.
When the second batch was released, there was an uproar on that forum complaining that they could no longer work on the hits. These workers were saying that he must have been hit by "scammers". They did not realize they were the "scammers" because they were not compleating the hits the way the requester intended. This is going to lead to a new blog post where I will go into more detail, but for now, I mostly agree with you FDR. With the exception that if the pay was better, you had better believe the master workers would have worked on these hits. A requester cannot expect to get the "so called best" workers to work for $6 or less an hour.
I agree with most things you wrote. This post was an analysis of the speed of Masters HITs. Obviously, if the pay was faster, they may have gone faster, but not incredibly faster I don't think. In any case, a custom qualified HIT will always go faster than a Masters HIT because it opens it to more people, Masters or not. That's just simple logic. This case really interested me, which is why I graphed the data. I believe that it is a little skewed, but I do not think it is horribly skewed.
DeleteI realize that a lot of people on that site are scammers, whether they admit that their work tactics are scamming or not, which is why I do not often post there. I did not know that detail about the HITs, but it does not surprise me.
Thanks for your feedback :)
There is a lot more to the story...
DeleteWhen he posted the hits with the masters qualification, those workers felt entitled to work for him, they email bombed him with suggestions and complaints.
As you noticed, between the first batch and his second batch with masters, he modified the instructions stating that anyone putting n/a was going to be manually reviewed.
When the third batch was posted w/o the masters qualification but with the new modified instructions and 5000 approved and approval percentage greater than 99%, a lot of the new workers were excluded and the talk on the forum was of people taking up to three minutes per hit. This effectively reduced the pay to about $2 an hour.
He is now getting acceptable results, the hits are getting completed for some reason, and the scammers have been shut down.
It is a requester win, worker lose situation. IMO his hits suck. Any email farming hits suck. His pay should be two to three times more for the work he is requiring.
I updated my text to fit some of what you talk about here.
DeleteThanks! I like being proven wrong. ;)
I would not say you were wrong, you just were not there real time watching as it developed. To my knowledge there was no discussion on TN about this hit. Reason being that email farming hits usually are low pay and those workers are not going to touch them.
ReplyDeleteThe changing dynamic on mturkforum has shifted away from Asian workers and become more workers that are new to turking who are impatient and racing to try to hit goals without learning how to turk smarter. Over time I see that dynamic changing as workers start to value their time more and stop working for requester who pay pennies per hour. Either way, great article, thanks for listening.
Chapter Foe, so-called "Master" workers are no different than any other workers. You've bought the deception of Amazon's "Master" qualification sales-pitch.
ReplyDeleteMany have high rejection rates, many have been soft and/or hard blocked.
Chapter Foe, so-called "Master" workers are no different than any other workers. You've bought the deception of Amazon's "Master" qualification sales-pitch.
ReplyDeleteThere are "Master" qualified people who have high rejection rates, and have been soft blocked, and or hard blocked by job posters.
Meanwhile, people who have completed tens of thousands of jobs, have few rejections or none, and 0 soft or hard blocks, but Amazon has not applied the label "Master" qualified to their accounts.
As a job poster, you're paying Amazon 2/3 the cost of a job because you really believe Amazon's bull-crap story about a "Master" qualified label.
We, MTurk workers will be always happy until we-pay-you-fast, transcriptions works and others surveys are there in Mturk. I also completed more than 11500+ HIT's and having an approval rate 98.7%. But till now I really search how to become as a Master worker. And I got lots of details from this posts. Lastly, I got some thing from this, that Masters works more daily than that of usual workers in Mturk. So that they got more rejection rates or some times more approval rates.
ReplyDeleteAdmin sir, can you please send me some links for working tutorials for new MTurk HIT's, those are posted daily?
Thank you.
This comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDelete