Date of Award
2024
Degree Type
Open Access Dissertation
Degree Name
Psychology, PhD
Advisor/Supervisor/Committee Chair
Tarek Azzam
Dissertation or Thesis Committee Member
Stewart Donaldson
Dissertation or Thesis Committee Member
David Fetterman
Dissertation or Thesis Committee Member
Thomas Archi
Terms of Use & License Information
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.
Rights Information
© 2024 Sarah Douville
Keywords
conceptual model, data visualization, evaluation, evaluation use
Subject Categories
Educational Assessment, Evaluation, and Research | Psychology
Abstract
Data visualization (data viz) is a valuable tool within evaluation for its ability to aid cognitive efficiency over text-based presentation of data (Card et. al., 1999; Evergreen, 2017, 2018; Few, 2012; Nussbaumer Knaflic, 2015; Tufte 2001). This exploratory multi-phase mixed methods research study considers purposes for using data viz in evaluation that can be achieved with that increased efficiency through the research question: “What conceptualizations of data viz use do program evaluators have beyond increased efficiency?”
In Phase I, secondary analysis of existing interview data with experts in both data viz and evaluation was used to better understand conceptualizations of data viz and their prevalence in evaluation. Support within evaluation was established for a Utilization-Focused Evaluation Framework (U-FE) (Patton & Campbell-Patton, 2022) conceptualization of data viz; an explain-explore model that considers data visualization from the perspective of who is having the experience along a continuum between explain and explore (Evergreen & Metzner, 2013; Kirk, 2019), and a model on a continuum from data to insight familiar to the fields of computer science and cognitive science that converts data into information, knowledge, understanding, sense-making, and/or insight (Chen, 2009).
A model of using data viz for stakeholder or audience engagement and to extend evaluation use emerged and was further described using follow-up interviews in Phase II. This audience engagement model resembles the information processing model of memory (Huang et al., 2009) from an evaluation perspective. In this model, attracting and holding audience attention is intended to lead to connection (interaction) and memory (learning), which in turn leads to evaluation use. The model also considers evaluation specific conceptualizations of the role(s) that brand identity, data viz design principles, artifacts, capacity building, professionalism, credibility, satisfaction, and confidence may play in audience engagement and evaluation use.
In Phase III, three of these conceptual frameworks (explain-explore, data to insight, and audience engagement) were presented as brief explainer videos to a sample of 131 evaluators who are members of the American Evaluation Association to determine their familiarity with and perceived usefulness of the models. Findings suggest that there is significant conceptual overlap between the models. All three models are complementary, appropriate in evaluation, add value to the efficiency rationale of data viz, make sense to evaluators, and are considered useful in evaluation. Each has the potential to benefit evaluators as they consider why they should use data visualization in their work and evaluators provided many examples of using each model in their work.
This research supported that program evaluators usually (38.2%) or always (42.7%) use data viz in their evaluation work, accept the efficiency rationale, and are interested in other reasons for using data viz beyond efficiency. While data viz is a time-consuming skill, providing evaluators with conceptualizations of data viz beyond efficiency may make them more willing to expend the time and effort needed to apply data viz to their evaluation work. Participant interest in both the content and the medium (e.g., brief explainer videos) suggests that there is interest, need, and desire for more professional development in data visualization and associated skills. Beyond skills workshops and “how to” guides, findings suggest a desire for more learning opportunities about abstract concepts, which offers new opportunities for teaching experiences and professional development opportunities within the profession.
Overall, findings suggest that the explain-explore model is a simple framework that an evaluator can use to consider the purpose of a particular visual before beginning to design and the data to insight model is a linear description of how to get the most information and insight out of a particular data viz. The audience engagement model is a holistic approach to thinking through the relationships in the evaluation to support evaluation use. While there is no clear hierarchy of models suggested in this study, comments supported that the audience engagement model is the most specific to evaluation – to the extent that it might not even be data viz specific.
ISBN
9798342762977
Recommended Citation
Douville, Sarah. (2024). Conceptualizations of Data Visualization Use Beyond Efficiency in Evaluation. CGU Theses & Dissertations, 865. https://scholarship.claremont.edu/cgu_etd/865.
