Wednesday, December 3, 2014

Amazon's Mechanical Turk workers protest: 'I am a human being, not an algorithm'

Amazon's Mechanical Turk workers protest: 'I am a human being, not an algorithm' 

Really good article on TheGuardian about Dynamo was published today. 

 Please contact for your HIT publishing needs.

Thursday, November 13, 2014

Reply to the Request for Work at $3 an Hour

Helping out academics who have been locked out of Amazon Mechanical Turk has been very rewarding recently. Not because of monetary gain,(we are not making minimum wage ourselves) but because we are behaving in an ethical manner and helping boost the Mturk marketplace to where it belongs. We have quite a bit of work and many academics believe in what we are doing. They are more than happy to pay a fair wage to receive excellent data.

But there are still others who are misinformed about Mturk. Using old data collected back in 2010 as a guide, I consistently see this outdated quote prestented...
Estimates of what workers can earn on these crowdsourced tasks range from about $1.20 to $5 an hour without any benefits. Employers treat them as independent contractors not covered by federal minimum-wage legislation.
 I have received a few requests from academics asking us (begging) to lower our payment to workers so they can afford posting their surveys using Mturk. This was my reply to one such request last night.

You are paying for time, not for how easy a survey is. When you are on your death bed, you cannot purchase more time, so paying workers a fair wage is essential to a successful marketplace that has been undermined by people who post underpaying work.
The reason a search of Mturk with the keyword "study" or "survey" yields so many results with underpaid work is because that work sits and sits until people who are bored or desperate complete it. The good work is taken almost immediately.
The survey we posted this morning was 120 responses, paid $3 and all data was collected in under an hour. On Monday we had 250 responses paying $1.50 and our data was collected in under an hour once again. Our criteria is workers with a 99% approval rating and over 10k hits approved.
We have very strict rules on who works for us and have less than a 4% rejection rate. We also have years of experience working with surveys and understand how to make them clear and understandable with as few worker problems as possible.

When it is said that it is costly for "us" academics, you probably mean yourself. Since our inception a couple of months ago we have published for 12 different clients and two are repeat customers now. I would suggest looking at this because the idea that going to mturk and throwing a few pennies to workers is outdated and unacceptable.
As you can see the bottom of the document is signed by quite a few "academics" for such a small project that has really just organized. There is also the Digital Labor Conference coming up that will add some insight into this discussion.

So for the tldr explanation
No, it is not up to the workers to decide, it is up to the requesters to behave in an ethical manner and pay a fair wage to start.

We will not publish underpaid work. There are dozens of requester middlemen available like CrowdFlower or CrowdSource that you can go to, but I guarantee they will charge you 3x our very affodable rate and you will end up paying workers $3 an hour.

Thanks for your consideration, but we have to decline.
If you value your research and you feel that it is important, then it is worth the money to receive quality results. If you do not have the funds to pay a proper wage, then you should not be posting work. 

I would like to suggest that all workers who read this and all academics who support this to sign the Guidelines for Academic Requesters and show that this way of thinking is no longer acceptable. Requesters who are paying workers $1-$3 an hour could be held accountable for their actions in the future. Just look at the CrowdFlower lawsuit that was just settled for more information. 

Wednesday, November 5, 2014

Amazon Accepted a Few International Masters

Amazon finally accepted a few international workers for their prized masters qualification yesterday. This is both good and bad. It is great that a few workers were given this qualification after completing hundreds of thousands of HITs, it is bad that along with those international workers, Amazon also accepted a bunch of worthless workers to water down this mystery qualification.

Here is some information on who just recieved the prized masters qualification along with some excellent international workers -

1) Turkers with 3,000 HITs completed got it. You can do that on day 11, and only because the first 10 days are restricted to 100 HITs per day.
2) Turkers with high 80 percent averages got it.
3) Today, people who don't even Turk anymore got it. Multiple people.

Requesters, save your money, do not spend that 30% fee on this useless qualification and set up your own custom qualifications for the standard 10% fee.

Wednesday, September 17, 2014

Turkers are People, not Labrats or Turkeys

It is disappointing to see that this is the way workers are viewed by some requesters.
With her research focused on sarcasm, I would hope that this was meant to be a joke of some kind, but it really comes off as derogatory and insulting. If her research is to be taken seriously and her studies are to be considered valid, I would think that she would want human beings to preform her tasks as opposed to "turkeys".

In the past couple of years we have published information to help both new and old requesters understand the workings of Mturk and Amazon. We have tried to give them an idea of what it is like to be a worker and how to ensure they receive quality results from a broken system that is Amazon Mechanical Turk.  Tweets like this just make it hard to move forward.

Thursday, August 28, 2014

The Ban on International Requesters on Mturk

We have started a website to help international requesters publish hits on Amazon Mechanical turk.

International requester hit publishing on amazon mechanical turk

Tuesday, August 19, 2014

How to Block Past Workers from Doing Surveys

This is an update about the current and easiest ways to ensure you recieve unique results for your surveys.

Republishing credit to Spamgirl from Turkernation

How to block past workers from doing your new HITs (API & GUI instructions) 

 Through the API:
Through the RUI/GUI

 1. Create a Qualification

Go here.

Click on .

Friendly Name: I'd recommend using your requester name, date and HIT type. For example, Spamgirl 12.8.13 Block Past Workers
Description: Describe the work this qualification entitles the Worker to do. For example, Workers with this qualification have already done my HITs before, so I'm using this qualification instead of blocking.

2. Assign Qualifications

Go here.

Click on the first Worker ID you'd like to give the qual to. Click .

Check the box next to your new Qualification, then click on .

Repeat for every Worker on your list.

3. Create your HITs

Go here.

Choose a template of your choice. Note that you cannot use your qualification with the Categorization or Sentiment templates as they don't allow you to change your qualifications, they force you to use the expensive Masters qualification. For categorization or sentiment HITs, use the Other custom template instead. Click Create Project.

The only settings which matter towards using your qualification are found when you click the Advanced >> link at the bottom right corner of the big gray area. Next, click the Worker Requirements >> link.

Worker Requirements: Choose Customize worker requirements.

Click Remove twice. On the last row, click Clear.

In the first drop down box, choose your qualification under Qualification Types you have created.
In the second drop down box, choose has not been granted.

Require qualification for preview: Yes.

Everything else is up to you.

Repeat step 2 every time your HITs are completed as you will need to give the qualification to all new people who do your HITs. Be sure to do this before reposting your HITs to ensure they can't retake it.


Why this is your best bet:

1) If a worker has to accept a HIT and then submit their ID, or try to remember if they've done it before (trust me, they won't), they'll get upset and your Turk Opticon rating will suffer.

2) Asking a worker to email you to find out if they've done it before is time consuming and more than any Turker will want to bother with for one HIT.

3) Putting a PUBLIC list of worker IDs online and asking Turkers to see if their ID is there IS A BIG NO-NO! This exposes their ID and all associated data, such as their wish list and any demographic information listed there, to the world. If you take this route, reputation damage will be swift and severe. On top of that, websites aren't 100% reliable, so they may not even be able to access the list.

4) Blocks put workers' accounts in jeopardy of suspension, so using them will totally ruin your reputation. Here is the hard block email text they receive when you block them:

Greetings from Mechanical Turk. We regret to inform you that you were blocked from working on HITs by the following requester(s): (That's YOU)

Requesters typically block Workers who submit poor quality work.

Requesters rely on Mechanical Turk for high quality work results. In order to maintain quality, we continuously monitor service activity. Should additional Requesters block you in the future, we may suspend your account. Please ensure your work quality is at a high standard. We encourage you to read the HIT instructions and follow them carefully.

We realize that this block may be a onetime occurrence for you. Should you maintain high work quality with no further complaints for the next few months we will dismiss this event.

Regards, The Mechanical Turk Team

If your survey will only be posted once, the best way to ensure Turkers only complete it once is to post a single HIT and specify how many workers you want to do it. For example, 1 HIT, 100 Turkers. This can be done through the GUI, Command Line Tools and API.

You can also use Qualtrics and other third-party services to do the work for you, but that will cost you money.

The best, easiest and most effective way to ensure no retakes is a qualification, though, so check out the link at the beginning of this post!


Thursday, July 24, 2014

Qualification Update on Amazon Mechanical Turk

Amazon announced some changes to qualification comparators. Once every ten years or so, Amazon makes an update to Mechanical Turk that benefits requesters.

There are three new qualification comparators available. The new Comparators, which include In, NotIn, and DoesNotExist, provide Requesters greater flexibility in targeting HITs to specific Workers. This will allow requesters to a specify locations without having to exclude every country in the world. For example, you can now set a qualification for “Location In US, CA, UK” instead of US only or not in IN, CK, TZ...

Read more about it on the Mechanical Turk blog.

Friday, June 20, 2014

Mturk No Longer Accepts International Requesters

Edit #2- Please read this new post on how to have your hits published using Amazon Mechanical Turk. BOOKMARK IT FOR NEWS AND INFORMATION 

Website is live,

Edit: 7/18/14
It has been confirmed that it is impossible for anyone who is not a US citizen to get an Amazon Payments account. 
There are no workarounds for this as stated in comments.

First Amazon banned new international workers, now Amazon has essentially banned all international requesters from Mturk.
This is the form letter that was sent this past week. . .

Dear Mechanical Turk Requester,

In connection with an update to our business practices, Mechanical Turk will require you to purchase prepaid HITs through Amazon Payments Personal or Business accounts. This requirement will go into effect on July 27, 2014.

Amazon Payments Personal accounts require you to provide your legal name, physical address, birth date and Social Security Number (SSN) to register. Amazon Payments Business accounts require you to provide your company name, physical address and company EIN to register. If you are registering for a Business account as a sole proprietor, you will be required to provide your date of birth and your Social Security Number instead of an EIN.

If you already have an Amazon Payments account, you can check the status of the account by visiting the following pages:
Personal -
Business *
You can create an Amazon Payments Personal or Business account by visiting these pages:
Personal -
Business -
Please note that the verification process may take 48 hours to complete.

If you would like to refund your Mechanical Turk balance or have additional questions, please contact customer service here:

Best regards,
The Mechanical Turk Team
And this is a screen cap from the requester website...

Without the social security number or EIN, international requesters will no longer be able to publish HITs. International companies can only publish hits if they have a US based address and an EIN. Requesters are required to pay workers using Amazon Payments. No other form of payment is allowed. If you are unfamiliar with Amazon Payments, that's no surprise. For years it was a side project that really never gained traction because other forms of payment were easier to use and accepted by a majority of online businesses. But now, Amazon Payments is a money making machine that Jeff Bezos is pushing into a war with PayPal for market share.With the new Amazon 3D smartphone scheduled to be released, Amazon is expanding into markets where they could make a huge impact and Amazon Payments is at the heart of it all.

This could be a major blow to Amazon Mechanical Turk. This is going to force international requesters into publishing HITs through a middleman. Academic requesters can contact US based universities and possibly reach some kind of an agreement with them to publish HITs, but for non-academic corporations, they will have to go through other channels. CrowdSource, Smartsheet and others will charge a fee on top of the Amazon standard fees to publish HITs. This means that international requesters will have to increase their budgets and workers will only receive a fraction of what the original payment was intended to be. Lining the pockets of middlemen is never a good business model. These middlemen do have something to offer though. They have the knowledge, experience and established presence that will help with a smooth Mechanical Turk experience, but it comes at a high price.

Another problem with this change is that Amazon is no longer the only player in the game. There are at dozens of crowdsourcing platforms available like Clickworker, microWorkers, oDesk and a host of others, all competing against Mturk (which is still in beta after 10 years.)

Is this the death of Mturk? Probably not, but it is another hinge on the coffin that is beginning to close.

Thursday, May 8, 2014

Effective Use of Amazon Mechanical Turk - Neeraj Kumar

Neeraj Kumar published  Effective Use of Amazon Mechanical Turk in May 2013 and updated in May 2014.

He posted a link to this blog in his paper and that is how I discovered it. It is an excellent paper discussing how to be a requester on Mechanical Turk and uncovers some of the pitfalls that many requesters face when trying to use Mturk.

This is a requester who "gets it" for the most part and there is a very helpful Q&A section at the bottom of the paper. Yet the focus of the HITs he published and the Q&A were academic hits. The data was not needed by a for-profit corporation where accuracy, speed and competency are required aspects of the data as opposed to being a strictly academic nature. They both deserve and require accurate results, but in an academic setting, having multiple people complete the same work is often an aspect of the research and is not required or needed by non-academic requesters.

I have written Neeraj with a link to this post and he might consider revising some of the issues I noticed in the Q&A section. There is a lot of excellent advice in this paper, but as he states, some of it may no longer be relevant.
I will quote the sections and respond.

Question 3

Question: I pick 1 cent per 5-second task which turns out to give a worker $7.20 per hour. Do you think it's appropriate?
Answer: If the task really takes 5 seconds on average, then 1 cent is slightly on the higher side of what people are usually paid [as of May 2013]. The usual solution is to add more than one 'job' per HIT, so that it takes a bit longer. Remember that you are being charged cost + 10%, or cost + 0.5 cents (whichever is higher) until you get to 5 cents per hit, you're paying proportionally more per HIT to amazon in fees. If your total volume is not very much (~$100), this doesn't matter, but at larger volumes, these costs can add up.
But in general, the thing to understand about costs on mturk is that they don't determine IF your job gets done, but rather WHEN your job gets done. More money == faster completion. It's hard to judge how much money is enough, so usually I start with the bare minimum, submit a small job (~20 HITs or so), and see how long it takes workers to complete that job. If it's too slow, I make the next batch a bit more expensive and repeat this process until I'm happy with the speed.
BTW, this kind of iteration is almost always needed, not just for pricing, but also to evaluate how well workers are completing the task (in terms of accuracy) and also for debugging.
Small problem with the question-  A 5 second task for a standard worker is not 5 seconds. Although the completion time may state 5 seconds, the worker has time involved in submitting the hit and accepting the next hit. So 5 seconds is really more like 8 seconds or $4.50 an hour, not $7.20. Also standard Mturk workers have to solve captcha questions every 25 hits to prove they are not automated bots completing hits, this further reduces their time and earnings.
Small problem with the answer - It is not hard to judge how much money is enough. This is a workplace and if you have real work that needs to be done, you should pay a fair wage to get it completed. If you are looking for simple transcription of business cards and only want Indian workers to complete the task, pay them a fair wage for India. If you want American workers to complete a task, you should start at the minimum wage in the United States and work up from there.
Requesters who view Mturk as a cheap labor force end up with poor results. There are hundreds of requesters on Mturk who are paying workers $2 an hour and less, but they are submitting the same hit 3, 4 or even 10 times in order to achieve a desired result. Why not pay one worker a fair wage to do the job right the first time?
The reason I point out these two countries is because many requesters do not know that India and America are the only two countries that pay workers in cash. The rest of the world is paid in Amazon currency which cannot be converted to cash. These are the countries where people earn a living off of Mturk and they should be paid a fair wage.  There is a huge workforce of excellent workers on Mturk who will not even consider working for less than $10 and up an hour. When you know how to access these workers, you are guaranteed excellent results.

Question 5 
In general, if you design your tasks such that it is not trivially easy to cheat, then spammers are usually not a huge issue. It also helps if your jobs are very small and cheap, as then there is often less incentive for spammers to devote time and resources to figure out how to cheat. ....
A simpler alternative is to just ratchet up the number of workers doing the same task. For simple things like attribute labeling, we required 3 responses per image, but for the face verification task, we had 10 responses each. I think we might also have thrown out outliers from this 10, but I'm not sure about that.
As stated in the article there are worker forums where workers communicate with each other. Most forums are honest and professional, but some are not. It is against Amazon's Terms of Service to automate any part of a HIT, but people write scripts that can auto complete many aspects of simple HIT design, then they share the scripts on unprofessional worker forums. Then, these scammers then work together and set there scripts to mark certain buttons and go full force on batches of HITs that are not designed properly. This defeats the purpose of having multiple workers complete the same HIT.
Using plurality to grade workers is wasting money for the requester and reducing pay for workers. It is unnecessary when you have qualified workers and proper hit design to weed out spammers and scammers.

So what I have seen on scammer forums is these workers feel justified in cheating because the pay is so low, why should they have to click every button or even do good work for a requester who does not value their time. On professional worker forums like Turkernation and Cloudmebaby, this type of behavior is frowned upon and users will be banned for even the slightest discussion of any type of cheating.

Question 6
Question: Can we have like a qualification test ourselves where each worker has to answer a couple of questions to make sure they don't just make a random guess. Do you think this is needed or is the standard 95% qualification option enough. Also if needed, does there exist such feature to do so on Amazon?

Answer: I would first try the simple 95% qualification before you move on to more sophisticated things. Run some smallish batches and see if the results look reasonable. It is possible to add custom qualification tasks, but I've never done them, so I don't know how they work. I think they also drastically cut down on the number of workers who are willing to do them, so only do it if it's absolutely necessary.
 In academic settings 95% is a grade of an "A"  but in the Amazon workforce, it should be regarded as an "F". There are only three reasons a worker has less than a 99% approval rating -
1. They are new workers and do not have a lot of hits completed. A few rejections can change their approval percentage drastically
2. They are foreign workers. These workers do not have as many "good" hits available to them and are forced to work for less than scrupulous requesters who use plurality to grade or just do not care about their workforce. Many of the best HITs are U.S. only.
3. They are cheaters, poor workers or outright scammers.

If using approval percentage alone it should be set to >98% to weed out the wrong workers. I understand that this question and the following questions were about a specific person asking about specific hits, but best practices should always be followed.

Question 8

In particular, the default worker qualifications (under 'Advanced' when creating a new job), now includes 'worker must be Master'. These are workers who have gone through a more stringent review process. While their work quality might be better (I don't have a good sense if this is actually true), this pool of workers is even smaller. So for very simple jobs (where there's little chance of screwing up), it's usually better to uncheck this option. Good replacement criteria are 'worker must have completed at least 1000 jobs' and 'worker approval rate >= 95%' (or thereabouts).
There are quite a few problems with using "master" workers.
You are limiting your pool of workers to a smaller workforce.
You are paying Amazon a premium of 30% per hit as opposed to the standard 10%
Amazon has no transparency on how this "masters qualification" is granted. Since the ban on international workers in 2013, there have been no new international master workers granted as well. If an international worker was not granted "master" status prior to 2013, they do not have a chance of receiving it now. The international pool of master workers is severely limited.

Using total approved HITs is irrelevant for the most part. Workers who are writers and survey takers primarily may take a couple of years to reach 1000 hits approved while other workers will routinely complete over 1000 hits a day.

Custom qualifications are the best way to get the best workers. If you are looking for results from the masses and do not care too much about quality then setting the standard qualifications to over 98% approved and 10,000 completed hits will get pretty good workers, but not the best. If you want the BEST workers and ACCURATE results there is really only one way to go...

1. Join a worker forum.
2. Discuss your HITs with workers prior to publishing anything. This ensures that your instructions can be understood, your workers know what to expect and that you can communicate any changes in instructions directly to your workforce.
3. Qualify workers, publish small batches and make sure that your workers understand the HITs and are completing them properly before releasing a large batch.
4. Pay them a fair wage. There are lots of requesters who pay over $20 per hour for the best workers, you are in competition with them. If you are publishing hits that need to be completed and are paying $8 an hour and one of the workers other requesters has hits up for $15 an hour, who do you think they will work for?
5. Try to set a schedule of when you will be releasing work so that your workforce can will know ot be available when the work is released.

New requesters thinking about using Amazon Mechanical Turk and looking in from the outside only see the negative part of Mturk. Requesters who have been around the block have learned some hard lessons with Amazon and know how to get the best workers and best results. A new requester sees the hits published by unethical requesters like infoscout and LinkedIn  paying less than $2 and thinks  this is the norm, but it is not. The HITs they cannot see are the ones that are being worked on day in and day out by the best that Mturk has to offer. These unethical pay practices have even led to lawsuits that could change the face of this industry and potentially bankrupt companies who are leading the way in these unethical practices.

 Edit: Neeraj has made some modifications to his paper to reflect some points in this post. Thanks!

Friday, February 21, 2014

Paper Published on Turkernation

Pretty interesting read from some academics about Mturk.
A little long, but worthwhile read.

Wednesday, February 5, 2014

The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

 Very insightful post from Spamgirl on Turkernation about the way Mturk is perceived by new requesters and the reality of what they are paying for.

Thursday, January 9, 2014

CrowdFlower -Dropping Mechanical Turk Helps Our Customers Get the Best Results

This is the post from the CrowdFlower Blog --  Commentary to follow

Dropping Mechanical Turk Helps Our Customers Get the Best Results

At CrowdFlower we continuously evaluate our partner channels to make sure our customers are successful using our platform. In December, we announced that Mechanical Turk would no longer be a partner channel. One of the primary reasons we did this was because we could no longer ensure that our customers would have the best experience using our platform and get the best results possible using the Mechanical Turk channel.
That said, we understand that some of our customers may be concerned with this change. To help assure customers that this would not adversely affect their jobs, but in fact improve them, we ran an experiment before we dropped the Mechanical Turk contributor channel. The experiment was designed to compare accuracy and throughput — the time it takes to complete a job.
How did we do it? We ran a job using the Mechanical Turk contributor channel only, and we ran the exact same job using all other partner channels and excluded Mechanical Turk contributors. The results were astounding: The non-Mechanical Turk contributors outperformed in both accuracy and throughput. We also ran the job a second time to the two different sets of contributors and added advanced quality control mechanisms. This time the job was limited to better-performing contributors in both channels, which we call Level 2 contributors, and it incorporated variable judgments. (Note: Selecting variable judgments means you can collect more judgments when contributors do not agree). Here are the results:
Job: Find Official Websites for a List of 1200 Companies

You will notice that there is no data for the job that we ran exclusively to Mechanical Turk Level 2 contributors. That’s because we were unable to get a single contributor to complete a task on the job.
The Bottom Line
Our customers will get better results, faster, using our other partner channels whose performance we continuously monitor. If you still have concerns about not being able to access Mechanical Turk contributors through our platform, please contact us so we can better understand and address your needs.

 I have to put in my two cents worth about this article because Crowdflower will not publish negative or honest comments on the blog and requesters/workers need to know the truth about what happened here.

CrowdFlower did not get any results because competent workers will not work for the low pay that Crowdflower offers.
When your CEO Lukas Biewald thinks that it is a public service to pay workers $2 per hour and class action lawsuits are filed against your company, you have serious problems.

These are some of the problems CrowdFlower had on Mturk with the hundreds of different types of HITs they published-
1. The interface was often broken or had coding errors
2. The price CrowdFlower charged customers for their middleman service was often 5x higher than what they were paying workers causing the work not to be completed. By the time CrowdFlower took it's 400% profit and 10% to Amazon, there was nothing left to pay honest and competent workers. These numbers are estimates because they varied from task to task.
3. Requesters started taking the time to learn how to use the Amazon API and avoid the exorbitant fees and pay workers directly
4. The unresponsive customer service from the worker forum and consistently poor quality control mechanisms caused CrowdFlower's reputation to be irreversibly damaged to a point where the best workers would avoid ALL of their tasks

There are excellent workers on Mturk, but they will not work for companies who do not pay fairly. I for one am glad to see CrowdFlower gone from Amazon. I hope that some of their customers will now drop Crowdflower and see them as the overpriced and incompetent middleman they are. Take some time to learn the Amazon API and start to design your own hits on Mturk.

Thursday, August 29, 2013

infoscout and Jon Brelig worst requester on mturk: What is Jon Brelig and Oscar Smith?

infoscout and Jon Brelig worst requester on mturk: What is Jon Brelig and Oscar Smith?: Experienced Mechanical Turk workers know who these unknown entities are and avoid them like the plague.     The copy business cards un...

 Interesting worker point of view found on the internet. I can't say that I agree with everything in the article, but it does show how your companies reputation can be damaged by not paying a fair wage.

Friday, March 8, 2013

DataJam LLC

How to destroy your requester reputation in under 24 hours.

This requester published a poorly designed hit that allowed submission without having all inputs checked. When the data came in incomplete, they rejected those hits and the negative reviews flooded in.

Here is where it gets bad for them, many turkers told them they were going to post negative reviews on TurkOpticon, so the requester thought it would be a good idea to post on there and communicate with workers en-mass.

This was their post.

 "Hello Everyone,
I am the engineer responsible for Data Jam's entries on mturk.

I want to address the many emails we've received regarding rejections on our most recent HIT Batch for "Classify Public Twitter Users' Profile Pics as Male or Female."

Each HIT was initially submitted with two assignments (requiring two turkers' response). We did a small initial batch of 400 HITs, of which we rejected 16, and approved the rest. Of 54,740 assignments we submitted in this batch, 2076 were automatically rejected earlier today for failing to "Answer all Prompts." In most cases, based on the view of the data Amazon has given us, only 8 of 9 images were successfully tagged when the user clicked Submit.

Many are claiming that they have proof that all images in their rejected submission were tagged. If someone can give me a screen shot, I can file a bug report with Amazon; because based on the data I am receiving, I have to believe they are mistaken. I myself am not at liberty to bend the rules attached with this assignment; it says specifically in the instructions that if 9 dots were not set, the submission would be automatically rejected. 97% of submissions did this successfully. I do not like rejecting anyone, having worked turk myself at one point; but rules are rules.

No submissions have been approved. After the trouble with this HIT Group, we've launched a second round, so that every HIT will now have a third turker respond to it. Afterward, I will automatically cross reference everyone's responses: the current plan is, assuming a responder did not routinely disagree with the other responders (>45% of the time), I will auto-approve responses. In practice so far, on average two people agree on the tag at least 75% of the time.

It is not cost effective to respond to each and every inquiry or complaint; we're trying to be dirt cheap here, and I'm frankly paid too well to spend time on email. Also I apologize that approval is taking so long; but we need all responses to arrive before we can cross reference responses to find lousy submissions (especially because we did not require the "Master" qualification).

Also, we realize the instructions are somewhat obtuse - if you're uncomfortable with it, please skip these HITs. The complicated matrix form was setup in an effort to streamline keyboard submissions, so that everything fit on the screen at once; in my own tests I was able to complete HITs in 15 seconds on average, which would have also yielded a substantially higher hourly average. I am not pleased that so many of you are working so hard and being paid so little. "

This was a great way to alienate workers and make yourself stand out in the turking community. Paying wokers under $4 an hour, telling them that you make too much money to care and then insulting them because they have a family to feed is never a good idea.
A couple of hours later the requester realized the damage they were doing and started reversing their rejections they had given that were because of their poor hit design, but the damage has already been done. They have since changed the statement above to this - -

Hello Everyone,

I am the requester in question.

For everyone who was upset about the rejections applied for the most recent Batch of "Classify Public Twitter Users' Profile Pics as Male or Female", thank you so much for bringing this to our attention. You all worked really hard on this HIT, and on further review it was obvious that the fault rested with the design of the HIT and not with your attempts to work it. Except for one or two responders we have identified as spammers, we are rolling back all the rejections associated with the HIT, and approving them and everything else that was pending.

Also, our target pay rate for this HIT was several times higher than what many of you achieved. We will try our best in the future to make our HITs less confusing and more streamlined, so that you can move faster and get paid better. This was our first big mturk roll out; we're still learning how to make good HITs for you.

(I can't figure out how to set the rating to "NO DATA", and agree its unfair that I rate myself, so I am marking myself 1 for everything. I will delete this review after this message has had a chance to circulate).

Thank you and have a nice day. 

That should have been the original message to workers, not the retraction.  Don't make the same mistake that DataJam did.
I would have to say that their reputation is teetering on the edge right now. Read the Lighting Buff post for some tips on how to protect your reputation and start off on the right foot on Mturk.

Thursday, February 7, 2013

How to Lose Money Using Mturk - The Myth of the Penny Hit

Why are you publishing hits that pay $0.01 per HIT? 

There are two reasons why you should not publish penny HITs.

First, from a requester perspective, there is absolutely no benefit to making a penny hit. Why? It is simple math and fees associated with Amazon publishing that make it a bad choice. Amazon Fees show that the standard payment for HIT completion is 10% of the hit value, but this does not apply for minimum pricing. Minimum pricing shows that for every HIT requesters are charged $0.005. So for your penny hit to be completed, you are going to pay a half cent or 50%! So what is the benefit of having a penny HIT when you are going to pay 5x more than the standard rate? There is no benefit. Even if you have a simple task that requires a single click for human judgement, you are going to lose out by making it a penny HIT.

From a worker perspective, the penny HIT is to be avoided. It is almost impossible to make a fair wage working on them and the risk to worker accounts does not justify the reward. Worker Perspective
Workers know that they can at best make $4.37 per hour, on average. But workers have overhead as well as requesters. Electricity, taxes, internet, computer, banking fees, etc. are all things that workers have to pay. This overhead further reduces the amount they can make working on penny HITs.

What is the solution?

It is really simple, put more data in your hits and pay workers more to reduce your cost of publishing. Instead of having one single radio button in your hit and paying a penny, put five radio buttons and pay a nickle. This will reduce your cost and allow you to publish more HITs.

This only addresses one area of the problem though, you are still not paying workers enough to work for you. Workers are smart, they will see what you have done and they KNOW why your hits are designed this way. There is a way around it so that you can pay workers fairly and still keep your costs within reason. Qualify your workforce so that only trusted workers can work for you.
You may wonder how a qualified workforce can SAVE you money, but this is also simple. When you can trust your workforce, you do not have to publish your hits multiple times to ensure accurate results. I know requesters with unqualified workforces who pay less than $3 to have their HITs completed. BUT they are paying for the same hit to be completed 6 times! Why not triple the pay to one qualified and trusted worker to ensure that you are getting accurate results? Your workforce is happy because you are paying fairly, your overhead is reduced by not overpaying Amazon and you are getting the results you hoped for when using Amazon Mechanical Turk.

Thursday, January 17, 2013

The Reasons Why Amazon Mechanical Turk No Longer Accepts International Turkers

As many of you may have heard, Amazon is not approving international Mturk accounts anymore. In 2012 Amazon put a hold on accepting new worker accounts for a couple of months. When they started accepting new workers again, it was only for workers in the United States.

There are three reasons for this:

Fraud - Amazon was having a difficult time verifying these accounts and a lot of fraud was occurring from outside the United States. Multiple accounts, sweatshops, money transfers and other Amazon TOS violations were occurring on these international accounts. Amazon would close the worker account and then they would simply open a new one. If you want to see some of this fraud, take a look at this forum- Romanian Mturk Forum. Over 300 pages long and 6000 posts. Almost every page of this forum has discussions on how to break Amazon's TOS. It is no longer active forum because of the halts on new international accounts.
Complaints - Requester complaints about the quality of work is the second reason. I personally know quite a few workers outside of the US that are excellent turkers and speak and write English better than many Americans. Because of the constant stream of complaints to Amazon about the poor quality of work from international IP addresses, something had to be done.  Since fraudulent workers outside of the US were not required to verify their accounts, they could just make a new account and get back to work under a different user ID.

Blocks - Blocks are a way Amazon to monitor worker accounts, although Amazon does not tell requesters this. When a worker account starts accumulating blocks from requesters, the workers account is suspended and reviewed by someone at Mturk. A requester could block a US worker and over time if that worker is not preforming well, they are removed completely from the worker pool and cannot make a new account. With the international accounts, a worker would simply change IP address and make a new account and set up a click script to automatically cheat HITs until blocked .  

Since Americans are verified using Social Security numbers, bank accounts, credit cards and addresses, American workers would protect and guard their accounts to ensure they are not suspended. This is NOT to say that international workers do not have a professional work ethic or are all out to cheat the system, it is just an additional restriction on American workers have to ensure that they can never make multiple accounts. 

Requesters are starting to take notice of this. The worker pool of cheap labor is getting smaller every day. Some requesters feel it is ethical to pay a worker $1-$2 for an hours worth of work. American workers, for the most part, will not work for this pay. New workers, international workers and naive turkers were the ones completing these tasks. Now, this substandard work is not being completed in a timely fashion anymore. 
For instance, CastingWords is now making their own interface because they need these international workers. 

So please, people in India, Pakistan and other foreign countries, please stop sending me emails asking me how to get your accounts verified. It is not going to happen. And to all the workers out there, protect your accounts no matter what country you are from.

Wednesday, January 9, 2013

Reverse Rejected Assignments in the Requester User Interface

Well, we are now into our 8th year of Mechanical Turk and there is now a way to reverse rejected assignments through the user interface. Amazon Mturk Blog Post About Reversing Rejections
Often, new requester will make a mistake and unintentionally reject assignments or grading errors cause work to be rejected. This is a very helpful tool to ensure that you can keep your workers happy if you make a mistake. This used to be a major hassle and required extensive contact with Amazon.

Last year @techlist provided a way to reverse rejections without having to contact Amazon. Now it has been added to the user interface.

Friday, December 28, 2012

Unfortunate Reality of Some Scamming Forums

EDIT - 1/14/2013 

Over the last week, I have noticed some substantial changes at mturkforum. The Admin has been activly deleting and removing posts that are violating requester rules and Amazon TOS. The members are policing their own forum and working to educate new members on what is allowed and what is not allowed. 
I cannot say that I believe in what they do or endorse them, but they are doing a fair job of getting rid of the users who are out to cheat the system which is what the original blog post was about. 
It takes time to build a positive reputation and the current forum members have years of negative rep to overcome. Keep up the good work and remember, I am not the only one watching you, the requesters who pay you are also watching.

ORIGINAL POST BELOW_________________________________

I was really hoping to never have to post this, but requesters should be aware about some issues with workers talking about their qualifications and the information within their hits.

It is best to try your best to configure your qualifications carefully so that workers who are out to cheat you cannot easily know what the actual qualification is.
This was a huge problem with the Paki-gamer forum, but now has become a huge problem for requesters. Every day there are discussions about the intricacies of HITs on this forum. How to work around qualifications, answers to qualifications, and answers to survey questions.
I personally tried to inform the people on this forum and the forum admin that they are breaking requester rules along with Amazon Terms of Service, but it fell upon deaf ears. I was even banned for trying to help. I wish that I could say something positive about this forum because some of the workers there are just misinformed or inexperienced and do not know they are breaking rules and cheating, but when I am banned from communicating, this is my only option to inform the requesters who are being cheated.

Every day there is a thread started on this forum called "Can't find good hits [current date]" in the Great hits section of the forum where the workers communicate with each other. In this thread is where most of the cheating happens.
This is an example Cheating post . This is a screen shot if they decide to take the post down. Screen shot

I really wish I could say this is an isolated incident, but it is not. My example may seem minor to many, but to this requester, it is not. It happens every day, day in day out all day long. When people inform the management about these issues, they get banned from the forum. It is in your best interest to carefully qualify your hits so that these workers are not the ones working on your hits and publicly sharing information that you do not want disclosed. .

If you are looking for a fairly specific group of workers and are finding some strange results within your HIT's, you might want to take a look at the scammer forum and see if those workers are cheating you.
If you want to communicate with legitimate workers who will not cheat you, then go to TurkerNation or Cloudmebaby.

Update: 1/7/13
To the honest workers on mturk forum - Andy does not care about the legitimacy of that forum. It is up to you to make that forum a legitimate and trustworthy resource because he will not. It really is your choice. If you want to stay there get rid of the cheaters.

I agree that the evidence that I posted is very limited, so over the next few days when I have time, I will do some screen captures of the daily rule breaking and cheating that I observe.  It really is quite easy to find, I just do not want to publish it because it perpetuates the cheating and can hurt the requesters involved. I will leave this link up for today, and post a new one tomorrow.
From 1/7/13
and another
 From 1/8/13
It is unfortunate that I can spend three minutes on this forum and find information like this. 
Today the discussion was on who could complete a survey the fastest, and then discussing missing attention checks. Great reason to have attention checks in all surveys.

Thursday, December 20, 2012

The Standard Problem of Pay on Mturk

I was reading a blog about election surveys using Mturk and ended up posting a fairly long comment in response to the authors.
The blog post did not focus on payment, but payment was a relevant issue in regards to results.
This is the link - The Monkey Cage 

As you can see, the focus is on the election survey results, not payment. But from the conclusions drawn, they did not get a representative sample of workers on Mturk because of the low payment and therefore had skewed results.
Hopefully you have seen from previous posts that there are high quality workers on Mturk who will give 110% to any requester who offers fair pay.

Take a look at the post and give your input. 

Thursday, November 15, 2012

Pakgamers Is Now DEAD! A Win For Requesters!

I was going to post a followup to Fredrick's post about Tim Edwards, but in doing some research I discovered that Pakgamers is now dead.
The final post on pakgamers

 As most of you know, I try to educate both workers and requesters on how to use Mturk so that it works out best for all of us. In doing this, I have to keep an eye on people who are out to scam Mturk. Since Amazon has put a halt on new accounts outside of the United States, one of the biggest scam forums for cheating Mturk has finally closed down. PAKGAMERS SCAMMER FORUM THREAD
If you read that entire thread, you will be shocked at the amount of dishonest practices that were going on overseas. Everything from how to make multiple accounts without getting caught to selling of American accounts overseas. This was a thorn in the side of honest workers for a long time and it is finally gone.

Now that Amazon Mechanical Turk is not accepting new accounts overseas, the quality of data is increasing dramatically. It is unfortunate because a lot of Asian workers were honest and hard working, but at the same time a majority of the accounts that were being created to cheat requesters were being created in India and Asia. I also need to state that there are many workers from outside the United States that have a better grasp on the English language than Americans. Many of them have a better work ethic and are diligent and honest workers.

For years the workers on TurkerNation have been reporting this forum to Amazon to show them what is going on and finally Amazon has taken steps to reduce this cheating and tighten the reigns on the workforce. This is not only a win for requesters on Amazon Mechanical Turk, it is also a win for workers. This adds an element of trust to the workforce that has never been possible in the past.

Add me on Twitter for updates to this blog. @pokernonsmoker

Wednesday, November 14, 2012

Thought Masters was just bad for non-blessed workers? It's even worse for requesters: the Tim Edwards story

It's a well known sentiment that the Masters qualification absolutely sucks for the majority of workers. Just to recap: Amazon pushes it on requesters, eliminating their chances at finding work, Amazon doesn't tell them how to get it, so even if they have one million HITs approved they might not be a “Master” and Amazon doesn't care at all about workers with any of these problems. Requesters can't even recommend workers as Masters, as a few have done for me. I have it on good authority that the Masters qualification is given to people who do random HITs on mTurk that Amazon puts up to test workers. I have no idea what these HITs pay, but if they pay little I doubt I've ever even noticed them. This really does explain why so many do not have Masters that deserve it. Take this how you will, a good journalist never names his sources. ;)

As a worker and requester (and consultant to requesters), though, I hate the Masters program doubly because as bad as it is for workers, it is even worse for requesters. Why?

  • Requesters need to pay three times more to use Masters. Instead of paying 10¢ on every dollar, they pay 30¢.
  • If a requester has a worker who they already know produces good work but that worker does not have Masters, that worker is barred from all of their HITs.
  • Since there are a lot less Masters than Amazon says there are, work completed by Masters is done at a slower rate than that of work done by workers with sane custom qualification schemes or workers with system qualifications.

Well, points one and two are very well proven, but I get a lot of flack when I try to claim point three. Fortunately, Tim Edwards came to my aid.

Thanks to a lone spammer or several spammers, Mr. Edwards decided that he had had enough and that he was going to give Masters a shot. Amazon has been pushing it hard very recently, so it may have seemed like a wise thing to do. Unfortunately for Mr. Edwards, he might not have known a few things:

  • He has several large datasets of workers. He could very easily have picked out hundreds of workers, the real Masters of his HITs, and qualified them with a custom qualification. I would have helped him out with this for as little as $30.
  • There aren't as many Masters as Amazon shows on the worker site, as I explained in my previous post. This could change, but at the moment, there are around 5,000 of them. I have no idea how many of those are active accounts, or if that is even a real number.

Anyway, on to the fun part. Opportunities like this don't come around very often, and reeling from some very new workers being granted Masters while some of their most loyal fans are left wondering where they went wrong, I decided to take down the number remaining of Edward's HITs every fifteen minutes. Here are my results, with fun annotations!

Added 11/16: As ChapterFoe points out in the comments, these results, as shocking as they are, might not be conclusive. I've left this post alone for history's sake — but Masters are very unlikely to do do HITs that pay little (which I confirmed) so I may not have proved much by posting this at all. In any case, it is an interesting look at work done on Mechanical Turk. I still think work done by Masters will always be slower, but this level of slower might be pushing it and is not representative of the whole body of work done by Masters.

Hey, don't shoot the messenger!

I'll post the full data once Edwards gets all his HITs back, but let me clue you in on some fun things right here and now:

  • With 78000 HITs at 10¢, Edwards was spending 20% more than necessary, a whooping $1,560. For that price, I would have run his HIT ten times over.
  • In the entire time Edwards was using Masters (twelve hours almost exactly), he got around 287 HITs done. If we figure each HIT takes one minute, using the crowd was actually 4.7x slower than if he had done all the work himself.
  • For every hour of Masters HITs (he got around 24 HITs/hr with Masters), Edwards gets 1,610 by letting those with the mark of Cain (I kid!) work on his HITs. That's 67 times more work being done!

As to the Masters program? It's bad for workers and bad for requesters and needs some serious reforms. I think it's a good idea at heart, but Amazon's lack of transparency on the issue dooms it to blog posts like this one.

Until next time!

Friday, November 2, 2012

HITs not getting completed? You might have missed Amazon's new default Masters requirement. As they intended.

Today, in their infinite wisdom and right before the weekend so they could deal with all complaints on Monday, the Mechanical Turk team decided to roll out a fundamental changes to the Requester Web UI.
Specifically, they decided that choosing which workers you want to work on your HITs is now an "Advanced" option, much like choosing paper or plastic at the grocery store is an advanced option or deciding whether you want to buy brand name or generic medicine is for advanced users only. Of course, the things that are really advanced are woefully poorly described and left out of the Requester UI, but we all knew that already.
This correlated with a big change on the Worker's site: the Categorization Masters and Photo Moderation Masters were combined into one big group, which, much to the surprise of everyone, led to 20,686 "Masters"!

Mechanical Turk is lacking many things. Math, unfortunately, is one of them.

So, how does this all play into the requester UI? Amazon, the sleeping giant, decided that instead of policing their site and helping requesters get better quality work by making it easier to create custom qualifications, they just wanted more money. A noble goal (by some definitions of noble), and to achieve it, they made Masters the default option for all new projects, and hid how to qualify workers:

Eh, worked for Microsoft.

Doesn't it just feel wrong to select "Customize Worker Requirements"? You're basically being told your results will be poor quality. I've been looking at the mTurk worker site, and I've noticed that some poor university students who run HITs have been oblivious to the change, and are likely about to run over their budgets and thrown out of school, all the while wondering why their results were so slow to be delivered:

Calling all Masters who are grandparents! Yes, all five of you, get up here!

So, be aware that this has changed when posting new HITs with the web UI. If you use the mTurk API or CLT, this doesn't affect you. 

As to how workers are taking it, I have a 99.9% approval rating and 22,000 approved HITs and am seriously looking into finding another way to make ends meet. There's nothing like not being able to get a license that not even the issuing body will tell you the requirements for, and quite frankly I'm quickly becoming sick of it. This sleight of hand with the Requester UI is a dirty trick that will confuse requesters and do little else.
Even if you win the rat race, you're still a rat. ~ William Sloane Coffin
Until next time!

Thursday, September 27, 2012

Academic Requesters: How to prevent retakes on your surveys without risking worker accounts

EDIT: Current information is on THIS POST. The information below is dated and might not work.

Preventing retakes on your survey is the most common issue that requesters have with mTurk. This guide explains an easy way to prevent retakes just by using the Amazon mTurk Command Line Tools (CLT) that work on Windows, OS X and GNU/Linux.

Note about Amazon's Block Method

A Google search will yield this forum post where an mTurk representative tells requesters to use the block worker function to prevent survey retakes. Unfortunately, Amazon's system is not perfect, so even if you do this a worker may still get the following form email:
Greetings from Mechanical Turk.
We regret to inform you that you were blocked from working on HITs by the following requester(s):
Requesters typically block Workers who submit poor quality work.
Requesters rely on Mechanical Turk for high quality work results. In order to maintain quality, we continuously monitor service activity. Should additional Requesters block you in the future, we may suspend your account. Please ensure your work quality is at a high standard. We encourage you to read the HIT instructions and follow them carefully.
We realize that this block may be a onetime occurrence for you. Should you maintain high work quality with no further complaints for the next few months we will dismiss this event.
The Mechanical Turk Team
Not only that, if you want to give our multiple surveys of the same type workers who took them before won't be able to take a different kind of survey from you.
IsaacM says this should never happen but experience begs to differ as requesters who use this method still get these emails sent out to workers. (There are many posts on the mTurk worker forum to this effect.) If there is a "glitch" in mTurk workers are going to be emailing you pleading for an unblock and you will get a bad reputation which may make responses from future surveys come more slowly or not at all.

Preferred Method: Qualifications

Instead of potentially causing a lot of trouble, you can use mTurk's qualifications to keep workers out of surveys. Basically, how it works is this: Your survey has a qualification pre-attached to it called "Did my survey" and it requires a value of 0 from a worker, a worker requests the qualification and it is auto-granted to them at a value of 0, and once they take the survey the qualification is incremented by 1.
First, create a file that looks like this, and name it something like
name:No retakes please!
description:Prevent retakes on my survey
keywords:prevent, retakes
To make the qualification, execute this command with the mTurk Command Line Tools (note: In this post, I've used the Windows syntax for all command line examples. For OSX/Linux, prepend the characters ./ to the beginning of the command and append .sh to the end of the first word, e.g.
createQualificationType -properties
You will get a QualTypeID printed to standard output as well as to a file called noretakes.success. You need this ID to change the values later.
Make sure to add this qualification to your file if you are making a new HIT with the CLT or to the necessary qualifications in the Hosted Requester GUI. Remember that the value should be 0.
Once the first run of the survey is complete, you can now raise everyone's values.
Whether you used the getResults command line tool or the web UI to get your results file, you should still have all of the work ID's who submitted work to your survey. Create a tab-delimited text file with the columns workerid and score. A few programs can create these files, such as Microsoft Excel and LibreOffice Calc. In Excel, they are called .tsv (tab separated values) files while in Calc they are saved as .csv but with a different delimiter (namely, tab or \t).
Your file should look something like this, with the symbol → representing a tab:
Then, run the following command to update everyone's qual score (note, the -qualtypeid parameter takes the QualTypeID generated earlier and stored in no_retakes.success)
updateQualificationScore -qualtypeid TPREVENTRETAKESQUALIDEXAMPLE -input noretakes.tsv

Harder method: Internal lists

A harder method to prevent retakes is to ask for a worker's work ID when they begin your survey, and if it's found on a list, tell them to return the HIT. Doing that is far out of the scope of this article, though, and this method is better as it will prevent any workers from even accepting a survey they cannot do, which will leave them on the site longer for people who can do them!
I hope this was useful for you and I hope your survey gets lots of replies. :) Remember to pay workers fairly! (at least 12 cents per minute)

Wednesday, September 26, 2012

Do not threaten workers in your instructions

This was the title of a thread on TurkerNation. The screen cap pretty much says it all. This turns off workers.

Post from turkernation used with permission

 Let me explain a little further so you can understand why this is a pretty big deal for workers.

Rejections- All workers get them eventually.  Workers understand that sometimes this happens, but they also know that any rejection will impact their approval percentage. Approval percentage is one of the most widely used qualifications that requesters use to find qualified workers. The best and most experienced workers do their best to keep this number above 99%.
For a new worker on Mturk, a few rejections can impact their approval percentage greatly. For every one rejection, a worker has to have 100 approvals to get their approval percentage back above 99%.
Most experienced turkers are in the 99.5-99.9% approval range, so a single rejection means these workers will have to complete 500-1000  HITs to remove the effect of one rejection.
This is the workforce that you want working on your HITs. They have the highest accuracy and have usually completed a wide variety of different HITs that give them the experience you need to work on any task you have in mind. If you turn them off in the first few sentences of your instructions, you will not be pleased with your results.

Blocks- Blocks are a big no no. ONLY use blocks when you are positive that a worker was blatantly trying to cheat or steal from you. Amazon does not go out of their way to tell requesters how big of a deal blocks are. When a worker is blocked, this can trigger Amazon to do a review of a workers account.
This is the sequence of events when this happens -
Day 1 - Amazon receives notice of block on worker account
Day 2 - Worker account is automatically suspended (cannot work or earn money)
Day 3-7 or more - Amazon reviews account.
Week or two later - Amazon makes a decision to permanently ban worker or reinstate account.

This is possibly a week or two of not being able to work or earn a living. So the instructions state "Note: we will review all work and reject any work that we find unsatisfactory. Workers whose work is rejected will also be blocked" Although that is what is written, the worker sees this, "we could block you and you might lose your Mturk account."

I am not saying that a requester should pay for work that is not done properly. If you have clear instructions, have a well designed HIT, and a worker does not submit usable work, you are within your rights to reject. BUT, you are better off forming some sort of communication with your workers and publishing in small batches with your qualified workforce prior to rejecting dozens of HITs. If you are going to block, use blocks to rid Mturk of the scamming workers who try to undermine the workplace and not to punish honest workers who may have made a mistake.

Below is the worker's view of how this should be handled....
Used with permission of author