Effects of a Teacher-Student-AI Agent Collaborative Model on Oral English Proficiency and Learning Motivation in Vocational Cruise English: A Quasi-Experimental Study with Triangulated Evidence

Abstract

This quasi-experimental study investigates the effectiveness of a Teacher–Student–AI Agent (TSA) collaborative teaching model on oral English proficiency and learning motivation among vocational college students in a cruise service English course. The TSA model, grounded in symbiotic theory and implemented through the Chaoxing AI Agent platform, integrates three subjects (teacher, student, AI agent) across four instructional stages (pre-class diagnosis, in-class interaction, post-class evaluation, and continuous optimization). A total of 186 students from four intact classes at a higher vocational college in Jiangsu, China, participated in the study, with two classes assigned to the experimental group (n = 93) and two to the control group (n = 93) over a 16-week semester. The experimental group received instruction integrating the Chaoxing AI Agent’s four core functions—AI Teaching Assistant, Customized AI Agent, AI Practice, and AI Learning Analytics—while the control group received traditional instruction. A triangulated evidence chain comprising platform behavioral logs, AI-generated instant scoring, and human-rated oral performance was employed. Pre- and post-test oral assessments were scored by both the AI system and two independent human raters, with inter-rater reliability (ICC) computed to examine AI scoring validity. Quantitative results indicated that the experimental group significantly outperformed the control group in oral fluency, accuracy, and communicative willingness (all p < .05), with effect sizes ranging from moderate to large (Cohen’s d = 0.62–1.18). Learning motivation and course satisfaction were also significantly higher in the experimental group (p < .001, d = 0.91). AI-human scoring agreement reached an ICC of 0.78 (95% CI [0.72, 0.83]), supporting the acceptability of AI-assisted assessment. Teacher interviews revealed enhanced instructional precision and data-informed pedagogical adjustments. The findings provide empirical evidence for the practical value of the TSA collaborative model and the Chaoxing AI Agent platform in vocational ESP instruction, offering an operational framework for AI-empowered teaching in similar contexts.