New attrition considerations when publishing longitudinal studies on Mturk

New attrition considerations when publishing longitudinal academic studies on Mturk. @amazonmturk is weeding out "nefarious participants" on mturk and it is causing larger attrition rates on academic studies. We lost 6% of our pool over three weeks from removed participants. We lost a little less than 1% of our participants from a screening qualification with a two day separation between qualification and study. These numbers used to be negligible. Maybe three or four out of 1000, but is now increasing at staggering rates.

For a study we published this week the researcher lost around $40 in financial cost, but in terms of data loss of potential responses, the cost is higher. These 20 participants preformed well on the first study and were invited back to the second part of the study. So right when we publish we are down 20 participants from our 330 data set reducing our return rate significantly. Right off the bat we are now looking at 85% of 310 instead of 330. From our point of view there is no reason for a 99% approval participant from the USA with over 1000 HITs completed to be removed from our data set. Amazon should know in 1000 HITs completed if they have a problematic worker.

Some requesters may look at this as a positive outcome of the BOT panic a year ago but we do not. We knew these were just poor quality workers in combination with requesters using insufficient qualifications. The fake bot panic might have been a kick in the pants for Amazon to remove some of these bad actors and clean up the marketplace or at least monitor it more closely. From our point of view, there was not a problem to begin with.

How do you know a participant was removed from Mturk?
When sending your email to their worker ID, the API will return something like this  
            "NotifyWorkersFailureCode": "HardFailure",
            "NotifyWorkersFailureMessage": "Cannot send notification to non-active end points",
            "WorkerId": "A2SVLXXXXXXXXX"

Non-active end points means the worker ID does not exist, but we know it did exist in the past because we have it in our CSV and database. 

Depending on the time between studies, the average return rate is from 85-90% from study to study. Time between the different parts of a study is a factor in return rate, but 85-90% used to be a reasonable average using quality participants. We have some longer gap longitudinal studies coming up in the next month or so and we will update this with new statistics.

EDIT : Our month publishing separation longitudinal study did not have a single removed participant.
The difference between the three week separation study and the month separation study was we were deep in the mturk pool on the three week study so there were many new mturk workers involved in it. Meaning we had already run the study with 4000 participants so they were not allowed to participate. With 4000 removed from the Mturk pool, many new "to Mturk" workers were participating and Amazon was actively deleting any nefarious ID's.