Concerns and Recommendations for Pain and Masculinity Research using Amazon Mechanical Turk: A Cautionary Tale

The goal of research is to discover and disseminate truth. The journey toward this goal can be complex as researchers are faced with decisions throughout the research process. In the field of psychology, errors during this process are prevalent which has contributed to the concerning replication crisis and cast doubt on the credibility of psychological research. Amazon’s Mechanical Turk (MTurk), a widely used online platform in psychology for collecting data, has contradictory evidence regarding its quality, which may be a significant factor in the propagation of errors. The quality of data collected through MTurk is significantly impacted by the strategies used to screen and clean data. To help researchers produce credible research, a cautionary tale is provided describing the potential consequences of adhering to the current standard of practice when collecting data through MTurk and the importance of examining data quality. The cautionary tale begins with a description of a theoretically grounded study in masculinity and pain that is argued to have compelling implications if examined effectively. The tale takes an unexpected and unfortunate turn as it is found that no matter how compelling or theoretically sound, poor research methods (e.g., data cleaning strategies) or poor-quality data can derail such a study and produce inaccurate results. The tale highlights flaws with the current standard of practice when collecting data through MTurk, analytical tools employed to identify the quality of the data as poor, the flawed results that could have been detected, and the problematic interpretations that might have been disseminated as well as recommendations for future research. The study highlights the complexity of the entire research process and the necessity for rigorous research practices to help ensure the effective discovery and dissemination of the truth in the field of psychology, particularly in masculinity and pain research using MTurk.