Using Attention Checks in Your Surveys May Harm Data Quality

Many researchers who are new to online studies ask us about using attention checks in their studies. Our view is that they can be helpful to eliminate random clickers, but should not be used as a guarantee for quality data.
A proper attention check needs to be straightforward and not ambiguous. They should not be designed to trick participants, but can require reading a block of text to ensure participants are reading instructions. We do not advise using memory checks as attention checks because they only gauges the participants memory, not if they are paying attention.

As far as Mturk goes, using high qualifications and time tested removal of problematic participants is the best way to ensure quality data.

Below is a link to a new article published on the Qualtrics blog about attention checks. It is a short and interesting read which we will continue to follow as they make advancements in this area.

https://www.qualtrics.com/blog/using-attention-checks-in-your-surveys-may-harm-data-quality/