Wednesday, September 12, 2012

How to unreject assignments on Mechanical Turk

My name is Fredrick Brennan, most people know me as "frb" on TN and I'm going to be handling the technical posts on the requester blog. :) This is my first one, and I hope someone somewhere gets use out of it.

If you reject assignments on Mechanical Turk, you probably get workers asking you to unreject them moments after you choose to. If their complaints are valid, you can really feel like you're in a tough spot. The mTurk Requester site is pretty crippled, and it doesn't include this function by default. Luckily, there are two ways to unreject them (provided you have enough money in your account first):

First unrejection method: Easiest

Techlist's Dahn Tamir maintains an unrejection page here. However, this page has been known to go down. Further, as of September 2012, it does not properly run over HTTPS, meaning that it transmits a requester's details in plaintext. Therefore, I highly recommend you download it yourself and run it on your own server. Guides for this can be found for Apache, lighttpd, and nginx.

Second unrejection method: Harder, but possibly more rewarding

The second way to unreject is to install the API yourself (here I'll demonstrate with the Perl API) and call the ApproveRejectedAssignment operation.

Here's an example Perl script you can use to achieve this (you'll need the Net::Amazon::MechanicalTurk CPAN module, install it with cpan -f -i Net::Amazon::MechanicalTurk. The -f "force" flag is not optional, it really is that shoddily written.)

#!/usr/bin/perl
#Thanks to Tani Hosokawa for the script!
use Net::Amazon::MechanicalTurk;
use Getopt::Long;
use strict;
my $sandbox = 0;
my %turk_options;
my $assignmentId;
GetOptions("assignmentId:s" => \$assignmentId, "sandbox" => \$sandbox);
if (not $sandbox) {
$turk_options{serviceUrl} = 'https://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester';
}
my $mturk = Net::Amazon::MechanicalTurk->new(
%turk_options
);
my $result = $mturk->ApproveRejectedAssignment(
Operation => "ApproveRejectedAssignment",
AssignmentId => $assignmentId,
);

Then, you'll want to get a text file together with the ID's of the assignments you want to unreject, which should look something like this:

A2xxxxxxxxxx
A2xxxxxxxxxx
A2xxxxxxxxxx
A2xxxxxxxxxx
A2xxxxxxxxxx
A2xxxxxxxxxx

You can either modify the above Perl script or just use a simple Bash script to run it through the assignments. In either case, make sure you use dos2unix file.txt beforehand to turn the \r\n characters into just \n characters if you made the file in Windows.

The final script will look something like (run it with ./unreject.sh filename):

dos2unix "$@"
for line in `cat "$@"`
do
perl unreject.pl --assignmentID=$line
echo $line
done

If you got this far, you're done. :D

7 comments:

  1. Excellent post FRB, thank you so much. It was essential that someone who has actually helped requesters reverse rejections write this post.

    ReplyDelete
  2. This script doesn't appear to work (anymore?) on Linux Mint. After forcing it to install with sudo cpan ..., I get this wonderful error:

    ➜ ~ perl unreject.pl --assignmentID ...
    defined(%hash) is deprecated at /usr/local/share/perl/5.14.2/Net/Amazon/MechanicalTurk/ModuleUtil.pm line 35.
    (Maybe you should just omit the defined()?)
    Can't locate object method "new" via package "Net::Amazon::MechanicalTurk::Transport::RESTTransport" at /usr/local/share/perl/5.14.2/Net/Amazon/MechanicalTurk/Transport.pm line 21.

    ReplyDelete
    Replies
    1. Just what it says, Maximus, you can remove the defined. This package is really old, but you can replace line 35 of /usr/local/share/perl/5.14.2/Net/Amazon/MechanicalTurk/ModuleUtil.pm with return %{*{"${package}::"}};

      As for the last error, I found a fix for it here: http://daveviner.blogspot.com/2009/12/amazon-mechanical-turk-perl-library.html

      I'm in the process of writing my own Python API for Mechanical Turk because I'm fed up with the current choices. I have it mostly working, but not ready for publishing yet.

      Thanks :)

      Delete
    2. Seems like the link in there is broken, but I found a CPAN module with the bug fixed from two years ago: http://search.cpan.org/~bdfoy/Net-Amazon-MechanicalTurk-1.01_01/

      Delete
  3. Though I made it through all of the steps, I get an error when trying to run it (./unreject.sh file.text):

    username$ ./unreject.sh file.text
    -bash: ./unreject.sh: Permission denied

    username$ sudo ./unreject.sh file.text
    sudo: ./unreject.sh: command not found

    ReplyDelete
  4. Ignore that comment (I don't see that I can delete it). I'm now receiving the same error as Maximus.

    ReplyDelete
    Replies
    1. Please see what I wrote to Maximus above. Thanks :)

      Delete