Date of Award
2025
Degree Type
Open Access Dissertation
Degree Name
Information Systems and Technology, PhD
Program
Center for Information Systems and Technology
Advisor/Supervisor/Committee Chair
Itamar Shabtai
Dissertation or Thesis Committee Member
June Hilton
Dissertation or Thesis Committee Member
Chinazunwa Uwaoma
Terms of Use & License Information
This work is licensed under a Creative Commons Attribution 4.0 License.
Rights Information
© 2025 Mohammed O Aljohani
Keywords
AI literacy, Large language models, Learning outcomes, Programming education, Prompt engineering
Subject Categories
Artificial Intelligence and Robotics | Educational Technology
Abstract
This study investigates the impact of large language model (LLM) usage, specifically ChatGPT, on student learning outcomes in programming education. The research adopts a mixed-methods approach, combining quantitative survey data from students and qualitative interviews with instructors. The study addresses three research questions: (1) the effect of LLM usage on undergraduate students' learning outcomes, (2) the influence of prompt engineering skills on this relationship, and (3) instructors' perceptions on these relationships. Quantitative data were collected from 159 students across two Saudi universities using a structured online survey with sections covering demographic information, LLM usage, self-reported programming understanding, and prompt engineering skills. Qualitative data were obtained through semi-structured interviews with programming instructors, covering LLM usage, prompt engineering skills, and their impact on student learning outcomes. The quantitative analysis utilized Partial Least Squares Structural Equation Modeling (PLS-SEM) to assess the measurement and structural models, including path coefficients, model explanatory power (R²), and predictive power (PLSpredict). Qualitative data were thematically analyzed using Atlas.ti to identify key themes related to instructor perspectives on the model. LLM usage positively impacts learning outcomes. While quantitative results did not show a significant moderating effect of prompt engineering skills, qualitative findings highlight its critical role in determining the positive effect of LLM usage on learning outcomes. The study emphasizes the importance of clear LLM usage policies and early prompt engineering training to promote meaningful engagement and maintain academic integrity in programming courses.
ISBN
9798291569931
Recommended Citation
Aljohani, Mohammed Owaidh. (2025). The Impact of LLMs Usage on Learning Outcomes for Software Development Students: A Focus on Prompt Engineering. CGU Theses & Dissertations, 1012. https://scholarship.claremont.edu/cgu_etd/1012.
