SUGILITE: Creating multimodal smartphone automation by demonstration

Toby Jia Jun Li, Amos Azaria, Brad A. Myers

نتاج البحث: فصل من :كتاب / تقرير / مؤتمرمنشور من مؤتمرمراجعة النظراء

131 اقتباسات (Scopus)

ملخص

SUGILITE is a new programming-by-demonstration (PBD) system that enables users to create automation on smartphones. SUGILITE uses Android's accessibility API to support automating arbitrary tasks in any Android app (or even across multiple apps). When the user gives verbal commands that SUGILITE does not know how to execute, the user can demonstrate by directly manipulating the regular apps' user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps' UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. Extensive error handling and context checking support forking the script when new situations are encountered, and provide robustness if the apps change their user interface. Our lab study suggests that users with little or no programming knowledge can successfully automate smartphone tasks using SUGILITE. Copyright is held by the owner/author(s). Publication rights licensed to ACM.

اللغة الأصليةالإنجليزيّة
عنوان منشور المضيفCHI 2017 - Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems
العنوان الفرعي لمنشور المضيفExplore, Innovate, Inspire
الصفحات6038-6049
عدد الصفحات12
رقم المعيار الدولي للكتب (الإلكتروني)9781450346559
المعرِّفات الرقمية للأشياء
حالة النشرنُشِر - 2 مايو 2017
الحدث2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017 - Denver, الولايات المتّحدة
المدة: ٦ مايو ٢٠١٧١١ مايو ٢٠١٧

سلسلة المنشورات

الاسمConference on Human Factors in Computing Systems - Proceedings
مستوى الصوت2017-May

!!Conference

!!Conference2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017
الدولة/الإقليمالولايات المتّحدة
المدينةDenver
المدة٦/٠٥/١٧١١/٠٥/١٧

بصمة

أدرس بدقة موضوعات البحث “SUGILITE: Creating multimodal smartphone automation by demonstration'. فهما يشكلان معًا بصمة فريدة.

قم بذكر هذا