CrowdFlower -Dropping Mechanical Turk Helps Our Customers Get the Best Results

This is the post from the CrowdFlower Blog --  Commentary to follow
LINK

Dropping Mechanical Turk Helps Our Customers Get the Best Results

At CrowdFlower we continuously evaluate our partner channels to make sure our customers are successful using our platform. In December, we announced that Mechanical Turk would no longer be a partner channel. One of the primary reasons we did this was because we could no longer ensure that our customers would have the best experience using our platform and get the best results possible using the Mechanical Turk channel.
That said, we understand that some of our customers may be concerned with this change. To help assure customers that this would not adversely affect their jobs, but in fact improve them, we ran an experiment before we dropped the Mechanical Turk contributor channel. The experiment was designed to compare accuracy and throughput — the time it takes to complete a job.
How did we do it? We ran a job using the Mechanical Turk contributor channel only, and we ran the exact same job using all other partner channels and excluded Mechanical Turk contributors. The results were astounding: The non-Mechanical Turk contributors outperformed in both accuracy and throughput. We also ran the job a second time to the two different sets of contributors and added advanced quality control mechanisms. This time the job was limited to better-performing contributors in both channels, which we call Level 2 contributors, and it incorporated variable judgments. (Note: Selecting variable judgments means you can collect more judgments when contributors do not agree). Here are the results:
Job: Find Official Websites for a List of 1200 Companies

You will notice that there is no data for the job that we ran exclusively to Mechanical Turk Level 2 contributors. That’s because we were unable to get a single contributor to complete a task on the job.

The Bottom Line
Our customers will get better results, faster, using our other partner channels whose performance we continuously monitor. If you still have concerns about not being able to access Mechanical Turk contributors through our platform, please contact us so we can better understand and address your needs.

 I have to put in my two cents worth about this article because Crowdflower will not publish negative or honest comments on the blog and requesters/workers need to know the truth about what happened here.

CrowdFlower did not get any results because competent workers will not work for the low pay that Crowdflower offers.
When your CEO Lukas Biewald thinks that it is a public service to pay workers $2 per hour and class action lawsuits are filed against your company, you have serious problems.

These are some of the problems CrowdFlower had on Mturk with the hundreds of different types of HITs they published-
1. The interface was often broken or had coding errors
2. The price CrowdFlower charged customers for their middleman service was often 5x higher than what they were paying workers causing the work not to be completed. By the time CrowdFlower took it's 400% profit and 10% to Amazon, there was nothing left to pay honest and competent workers. These numbers are estimates because they varied from task to task.
3. Requesters started taking the time to learn how to use the Amazon API and avoid the exorbitant fees and pay workers directly
4. The unresponsive customer service from the worker forum and consistently poor quality control mechanisms caused CrowdFlower's reputation to be irreversibly damaged to a point where the best workers would avoid ALL of their tasks

There are excellent workers on Mturk, but they will not work for companies who do not pay fairly. I for one am glad to see CrowdFlower gone from Amazon. I hope that some of their customers will now drop Crowdflower and see them as the overpriced and incompetent middleman they are. Take some time to learn the Amazon API and start to design your own hits on Mturk.