How pre-processing decisions affect the reliability and validity of the approach-avoidance task: Evidence from simulations and multiverse analyses with six datasets
Publication year
2024Number of pages
32 p.
Source
Behavior Research Methods, 56, 3, (2024), pp. 1551-1582ISSN
Publication type
Article / Letter to editor
Display more detailsDisplay less details
Organization
SW OZ BSI KLP
Journal title
Behavior Research Methods
Volume
vol. 56
Issue
iss. 3
Languages used
English (eng)
Page start
p. 1551
Page end
p. 1582
Subject
Experimental Psychopathology and TreatmentAbstract
Reaction time (RT) data are often pre-processed before analysis by rejecting outliers and errors and aggregating the data. In stimulus–response compatibility paradigms such as the approach-avoidance task (AAT), researchers often decide how to pre-process the data without an empirical basis, leading to the use of methods that may harm data quality. To provide this empirical basis, we investigated how different pre-processing methods affect the reliability and validity of the AAT. Our literature review revealed 108 unique pre-processing pipelines among 163 examined studies. Using empirical datasets, we found that validity and reliability were negatively affected by retaining error trials, by replacing error RTs with the mean RT plus a penalty, and by retaining outliers. In the relevant-feature AAT, bias scores were more reliable and valid if computed with D-scores; medians were less reliable and more unpredictable, while means were also less valid. Simulations revealed bias scores were likely to be less accurate if computed by contrasting a single aggregate of all compatible conditions with that of all incompatible conditions, rather than by contrasting separate averages per condition. We also found that multilevel model random effects were less reliable, valid, and stable, arguing against their use as bias scores. We call upon the field to drop these suboptimal practices to improve the psychometric properties of the AAT. We also call for similar investigations in related RT-based bias measures such as the implicit association task, as their commonly accepted pre-processing practices involve many of the aforementioned discouraged methods.
This item appears in the following Collection(s)
- Academic publications [246936]
- Electronic publications [134293]
- Faculty of Social Sciences [30577]
- Open Access publications [107816]
Upload full text
Use your RU credentials (u/z-number and password) to log in with SURFconext to upload a file for processing by the repository team.