Averaging Multiple Judges to Improve Validity: Aid to Planning Cost-Effective Clinical Research
Behavioral and Organizational Sciences (CGU), Psychology (Pomona)
Clinical Psychology | Psychology | Social and Behavioral Sciences
Frequently, a researcher has the option of having each S rated on a variable by several judges rather than by a single judge. A composite judge is formed by averaging the judgments of 2 or more individual judges. Composite judges are a valuable research tool because they decrease random error of measurement and increase validity. Data analyses show that (1) a simple equation, Equation 1, predicts extremely accurately the expected validity for any given composite size; (2) standardizing each judge's judgments is not necessarily important; (3) using a composite judge can yield very large validity gains when the redundancy of judges is low; and (4) there is a simple method for estimating confidence intervals around Equation 1's predictions. Two of Equation 1's applications in planning research (increasing the validity coefficient in a test validation study and attaining the desired level of statistical power) are discussed.
© 1990 American Psychological Association
Tsujimoto, R. N., Hamilton, M., and Berger, D. E. (1990). Averaging Multiple Judges to Improve Validity: Aid to Planning Cost-Effective Clinical Research. Psychological assessment: A journal of consulting and clinical psychology, 2(4), 432-437. doi: 10.1037/1040-3518.104.22.1682