Ruffle &Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System

Robin Schmucker, Meng Xia, Amos Azaria, Tom Mitchell

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    2 Scopus citations

    Abstract

    Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle &Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle &Riley’s ability to support biology lessons in two between-subject online user studies (N=200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle &Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle &Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.

    Original languageEnglish
    Title of host publicationArtificial Intelligence in Education - 25th International Conference, AIED 2024, Proceedings
    EditorsAndrew M. Olney, Irene-Angelica Chounta, Zitao Liu, Olga C. Santos, Ig Ibert Bittencourt
    PublisherSpringer Science and Business Media Deutschland GmbH
    Pages75-90
    Number of pages16
    ISBN (Print)9783031643019
    DOIs
    StatePublished - 2024
    Event25th International Conference on Artificial Intelligence in Education, AIED 2024 - Recife, Brazil
    Duration: 8 Jul 202412 Jul 2024

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume14829 LNAI
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349

    Conference

    Conference25th International Conference on Artificial Intelligence in Education, AIED 2024
    Country/TerritoryBrazil
    CityRecife
    Period8/07/2412/07/24

    Keywords

    • authoring tools
    • conversation analysis
    • conversational tutoring systems
    • intelligent tutoring systems
    • large language models

    Fingerprint

    Dive into the research topics of 'Ruffle &Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System'. Together they form a unique fingerprint.

    Cite this