Research Project

Origami Sensei: A Mixed Reality AI-Assistant for Creative Tasks

Keywords

AI-Assistant, Mixed Reality, Origami, Hands-on learning, Convolutional neural networks

Publications

Chen, Q., Mishra. R., Purnamasari, L., El-Zanfaly, D., Kitani, K. (2025). Origami Sensei: A Mixed Reality AI-Assistant. In CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26–May 01, 2025, Yokohama, Japan. ACM, New York, NY, USA, 18 pages. https://doi.org/10.1145/3706598.3714099

Chen, Q., Mishra, R., El-Zanfaly, D., & Kitani, K. (2023). Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands. Companion Publication of the 2023 ACM Designing Interactive Systems Conference, 147–151. https://doi.org/10.1145/3563703.3596625

Team

Qiyu Chen, Richa Mishra, and Lia Purnamasari advised by Kris Kitani and Dina El-Zanfaly

Description

Learning creative hands-on skills, like origami, can be difficult for beginners. Conventional instructional methods often fail to support the experiential aspect of learning with timely and personalized feedback. Despite recent advancement of AI and Extended Reality in many fields, there is a lack of research on supporting learning in creative hands-on tasks. We investigate an AI-augmented Mixed Reality approach for learning hands-on creative tasks by introducing Origami Sensei as an approach for learning origami. Origami Sensei identifies the current step and relative locations of the paper using origami detection models, and projects real-time, personalized instructions directly onto the paper. We conducted a user study (n=18) comparing it with traditional video tutorials. Our findings show that participants prefer Origami Sensei, and it increases task efficiency and learner engagement. We introduce design insights for developing AI-augmented MR systems and highlight the potential for extending this approach to other creative hands-on tasks.