Research Project

Mixed reality In-situ Welding

Integrating XR and tinyML to train skilled workers for future opportunities in creative arts and manufacturing

Keywords

Designing with, Future work force, Extended Reality, Tiny ML, Embodied Cognition, Tangible User Interfaces

Publications

Johnson, T., Chen, Z., Knowles, A., Li, A., Yi, S., Zhuang, Y., Byrne, D., & El-Zanfaly, D. (2024). XRweld: An In-Situ Extended Reality Platform for Welding Education. SIGGRAPH 2024, 28 July- 1 August, Denver, USA. (In Press)

Chen, Z., Johnson, T., Knowles, A., Li, A., Yi, S., Zhuang, Y., Byrne, D., & El-Zanfaly, D. (2024). Augmenting Embodied Learning in Welding Training: The Co-Design of an XR- and tinyML-Enabled Welding System for Creative Arts and Manufacturing Training. Proceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction, 1–14. https://doi.org/10.1145/3623509.3633398 (Honorable Mention)

Johnson, T., Li, A., Knowles, A., Chen, Z., Yi, S., Zhuang, Y., El-Zanfaly, D., & Byrne, D. (2023). Augmenting Welding Training: An XR Platform to Foster Muscle Memory and Mindfulness for Skills Development. Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces, 61–64. https://doi.org/10.1145/3626485.3626544 (Best Demo Paper Award)

Team

Dina El-Zanfaly, Daragh Byrne, Tate Johnson, Semina Yi, Yuehui, Ann Li, Zhenfang Chen, Lia Purnamasari, Tiger Tang, Franklin Xu, and Fanjie Jin in collaboration with The Industrial Arts Workshop (IAW)

Description

Hands-on skills and training are pressing needs for growing the American workforce. It has become increasingly challenging to attract and maintain novice welders to train. Additionally, welding training consumes a lot of resources including materials, energy and time. Hands-on skills are usually difficult to externalize and represent, and most current welding training methods use non-immersive simulations for skill acquisition.

In this research, we will develop and test a mixed-reality set-up that will enable in-situ training for stick welding. This set-up (see below) will provide interactive instructions and feedback to the trainees, help them achieve and explore creative structures and outcomes, and engage them in learning and staying within the training program. There is a need for such a training set-up for all ages and vocations. We partnered with the Industrial Arts Workshop (IAW), to develop, test, and deploy the in-situ training system. IAW focuses on training teenagers in the underserved community of Hazelwood through creative arts welding projects. This project will enable us in the future to democratize learning to transfer hands-on skills by creating immersive interfaces in many fields such as operating machines, woodworking, and physical computing, and to improve recruitment and retention of a skilled workforce.

We are currently conducting design Workshops and focus Groups with welding instructors and students to identify and synthesize the key focus areas, moments and actions of the instruction experience. It will also gather problems of practice for welding training, recruitment and retention as well as tasks and creative activities that meaningfully connect learners' authentic interests with skills development outcomes.

The In-situ mixed reality system set-up consists of two main components (A helmet integrating an Extended Reality (XR) headset, and a smart interactive virtual welding guide projected inside the XR headset.)

Other Projects