Date of Award
2025
Degree Type
Open Access Dissertation
Degree Name
Psychology, PhD
Program
School of Social Science, Politics, and Evaluation
Advisor/Supervisor/Committee Chair
Stewart I. Donaldson
Dissertation or Thesis Committee Member
J. Bradley Cousins
Dissertation or Thesis Committee Member
Michelle Sloper
Dissertation or Thesis Committee Member
Steffen Bohni Nielsen
Terms of Use & License Information
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.
Rights Information
© 2025 Ann Marie Castleman
Keywords
evaluation capacity, evaluation commissioner, international development, international nongovernmental organization, research on evaluation
Subject Categories
International Relations | Psychology
Abstract
Evaluation commissioners play a central role in evaluation practice by generating demand for evaluation through issuing calls for evaluation services and managing evaluation contracts (Lemire, Nielsen et al., 2018; Nielsen et al., 2018b), particularly in the international development sector. Yet few competency sets specify the knowledge and skills needed to conduct this work. Even less is known about the evaluation marketplace in the international development sector and the contract mechanisms used to procure evaluation services.
This research explored the capacity to commission evaluation among international nongovernmental organization (INGO) evaluation commissioners, including individual competencies and practices for commissioning an evaluation, factors that facilitate and constrain the commissioning process, and ways in which evaluation commissioners can contribute toward decolonizing evaluation. The research questions were answered using an explanatory sequential mixed methods design consisting of two phases: (a) a quantitative phase (survey) followed by (b) a qualitative (multiple case study) phase comparing and contrasting INGOs with emerging and high evaluation capacity.
Phase One of the research included a survey of INGO evaluation commissioners (N = 93) to understand individual and organizational evaluation capacity, commissioning practices and influential factors, and awareness of calls to decolonize evaluation. Respondents possessed a median of 10 years of commissioning experience and reported (on a five-point scale) high levels of individual evaluation capacity (Mdn = 4.5, IQR = 4.3, 4.6). Organizational evaluation capacity varied by organizational resources for evaluation and organizational use of evaluation. On average, respondents rated the INGO capacity to use evaluation (M = 3.7, SD = 0.64) slightly higher than organizational resources provided for evaluation (M = 3.4, SD = 0.75).
Commissioners reported always or often recruiting external evaluators by publicly posting requests for proposals (67.6%, n = 54), by circulating among a short-list of evaluators (33.8%, n = 27), or by referrals (30.1%, n = 24). Three-quarters of respondents (n = 62) reported seeking evaluation expertise as the top qualification when recruiting evaluators; yet two-thirds (n = 53) reported they did not use an evaluator competency framework when reviewing evaluator qualifications. Over 60% (n = 48) of respondents reported being moderately to very aware of calls to decolonize evaluation. However, 16.6% (n = 12) reported contributing a lot or a great deal to these efforts.
Phase Two consisted of a multiple case study of four INGOs with differing levels (emerging, high) of organizational evaluation capacity by evaluation resources and evaluation use. Interviews (N = 21) and document review (N = 36) were analyzed by case. A cross-case synthesis illuminated that the evaluation commissioning process is relatively standard across INGOs but that interpersonal, organizational, and systems-level factors influence commissioner decision making and the commissioning process.
Interpersonal factors included coordination and communication not only with evaluators but with internal and external interest holders (N = 4 cases). Organizational factors included organizational evaluation policies (N = 4 cases), organizational culture (n = 3 cases) and structure (n = 2 cases), and INGO staff evaluation capacity (n = 2 cases). Systems-level factors included the evaluation marketplace (N = 4 cases), donor requirements (n = 3 cases), and the nature of the development sector (n = 3 cases).
Commissioners had varying levels of awareness and agreement with calls to decolonize evaluation. Some articulated ways to contribute to decolonization by contracting with local evaluators, adopting participatory approaches to engage local communities, and using evaluation methodologies indigenous to the local context.
The collective findings illuminate the state of the field regarding evaluation commissioning, identify strengths and challenges in the commissioning process, and point to future directions for strengthening this capacity and contributing to decolonizing evaluation within the international development sector.
ISBN
9798293858156
Recommended Citation
Castleman, Ann Marie. (2025). The Power to Commission: Exploring the Evaluation Capacity and the Transformative Role of Evaluation Commissioners in the International Development Sector. CGU Theses & Dissertations, 1031. https://scholarship.claremont.edu/cgu_etd/1031.
