Exploratory Study on LLM-Assisted Evaluation of PMBOK-Based Argumentative Essays in a Master’s Programme (#2722)

Authors

Ormeno-Arriagada, Pablo

Jones, Eduardo

Abstract

The integration of professional standards such as the Project Management Book of Knowledge knowledge areas into graduate-level project management programmes has increased the need for robust and scalable assessment strategies. At the same time, Large Language Models have emerged as potential tools for supporting academic evaluation. This exploratory study examines the feasibility of using assisted assessment to evaluate argumentative essays structured around knowledge areas in a Master’s programme. A limited sample of student essays was analysed using predefined evaluation metrics focusing on conceptual integration, argumentative coherence, cross-domain synthesis, and alignment with knowledge area domains. Model-generated evaluations were examined to identify patterns in domain coverage, scoring consistency, and analytical depth. The study does not aim to validate automated grading but to explore methodological viability and identify strengths, limitations, and risks associated with AI-assisted evaluation in professional graduate education. Findings highlight both the potential of LLMs to detect structural alignment with the knowledge areas and the need for human oversight to ensure contextual and critical judgment. This pilot contributes to ongoing discussions on AI-supported assessment in higher education. The study analyzes essays written by postgraduate students in a PMI-aligned Master’s program and compares dimension-level scoring patterns between human evaluators and GPT-based evaluation.