Ruffle &Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System

Robin Schmucker, Meng Xia, Amos Azaria, Tom Mitchell

פרסום מחקרי: פרק בספר / בדוח / בכנספרסום בספר כנסביקורת עמיתים

1 ציטוט ‏(Scopus)

תקציר

Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle &Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle &Riley’s ability to support biology lessons in two between-subject online user studies (N=200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle &Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle &Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.

שפה מקוריתאנגלית
כותר פרסום המארחArtificial Intelligence in Education - 25th International Conference, AIED 2024, Proceedings
עורכיםAndrew M. Olney, Irene-Angelica Chounta, Zitao Liu, Olga C. Santos, Ig Ibert Bittencourt
מוציא לאורSpringer Science and Business Media Deutschland GmbH
עמודים75-90
מספר עמודים16
מסת"ב (מודפס)9783031643019
מזהי עצם דיגיטלי (DOIs)
סטטוס פרסוםפורסם - 2024
אירוע25th International Conference on Artificial Intelligence in Education, AIED 2024 - Recife, ברזיל
משך הזמן: 8 יולי 202412 יולי 2024

סדרות פרסומים

שםLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
כרך14829 LNAI
ISSN (מודפס)0302-9743
ISSN (אלקטרוני)1611-3349

כנס

כנס25th International Conference on Artificial Intelligence in Education, AIED 2024
מדינה/אזורברזיל
עירRecife
תקופה8/07/2412/07/24

טביעת אצבע

להלן מוצגים תחומי המחקר של הפרסום 'Ruffle &Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System'. יחד הם יוצרים טביעת אצבע ייחודית.

פורמט ציטוט ביבליוגרפי