Averaging Multiple Judges to Improve Validity: Aid to Planning Cost-Effective Clinical Research

Document Type

Article

Department

Behavioral and Organizational Sciences (CGU), Psychology (Pomona)

Publication Date

1990

Disciplines

Clinical Psychology | Psychology | Social and Behavioral Sciences

Abstract

Frequently, a researcher has the option of having each S rated on a variable by several judges rather than by a single judge. A composite judge is formed by averaging the judgments of 2 or more individual judges. Composite judges are a valuable research tool because they decrease random error of measurement and increase validity. Data analyses show that (1) a simple equation, Equation 1, predicts extremely accurately the expected validity for any given composite size; (2) standardizing each judge's judgments is not necessarily important; (3) using a composite judge can yield very large validity gains when the redundancy of judges is low; and (4) there is a simple method for estimating confidence intervals around Equation 1's predictions. Two of Equation 1's applications in planning research (increasing the validity coefficient in a test validation study and attaining the desired level of statistical power) are discussed.

Rights Information

© 1990 American Psychological Association

Terms of Use & License Information

Terms of Use for work posted in Scholarship@Claremont.

Link to Full Text

Share

COinS