TY - GEN
T1 - Ruffle &Riley
T2 - 25th International Conference on Artificial Intelligence in Education, AIED 2024
AU - Schmucker, Robin
AU - Xia, Meng
AU - Azaria, Amos
AU - Mitchell, Tom
N1 - Publisher Copyright:
© The Author(s) 2024.
PY - 2024
Y1 - 2024
N2 - Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle &Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle &Riley’s ability to support biology lessons in two between-subject online user studies (N=200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle &Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle &Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.
AB - Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle &Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle &Riley’s ability to support biology lessons in two between-subject online user studies (N=200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle &Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle &Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.
KW - authoring tools
KW - conversation analysis
KW - conversational tutoring systems
KW - intelligent tutoring systems
KW - large language models
UR - http://www.scopus.com/inward/record.url?scp=85200258936&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-64302-6_6
DO - 10.1007/978-3-031-64302-6_6
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:85200258936
SN - 9783031643019
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 75
EP - 90
BT - Artificial Intelligence in Education - 25th International Conference, AIED 2024, Proceedings
A2 - Olney, Andrew M.
A2 - Chounta, Irene-Angelica
A2 - Liu, Zitao
A2 - Santos, Olga C.
A2 - Bittencourt, Ig Ibert
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 8 July 2024 through 12 July 2024
ER -