Research Project

Mixed reality In-situ Welding

Integrating AR, tinyML and generative AI to train skilled workers for future opportunities in creative arts and manufacturing

Keywords

Designing with, Future work force, Extended reality, Tiny ML, Embodied Cognition, Tangible User Interfaces

Publication

Work in progress

Team

Dina El-Zanfaly, Daragh Byrne, Semina Yi, Yuehui, Ann Li, Zhenfang Chen, Fanjie Jin in collaboration with The Industrial Arts Workshop (IAW)

Description

Hands-on skills and training are pressing needs for growing the American workforce. It has become increasingly challenging to attract and maintain novice welders to train. Additionally, welding training consumes a lot of resources including materials, energy and time. Hands-on skills are usually difficult to externalize and represent, and most current welding training methods use non-immersive simulations for skill acquisition.

In this research, we will develop and test a mixed-reality set-up that will enable in-situ training for stick welding. This set-up (see below) will provide interactive instructions and feedback to the trainees, help them achieve and explore creative structures and outcomes, and engage them in learning and staying within the training program. There is a need for such a training set-up for all ages and vocations. We partnered with the Industrial Arts Workshop (IAW), to develop, test, and deploy the in-situ training system. IAW focuses on training teenagers in the underserved community of Hazelwood through creative arts welding projects. This project will enable us in the future to democratize learning to transfer hands-on skills by creating immersive interfaces in many fields such as operating machines, woodworking, and physical computing, and to improve recruitment and retention of a skilled workforce.

We are currently conducting design Workshops and focus Groups with welding instructors and students to identify and synthesize the key focus areas, moments and actions of the instruction experience. It will also gather problems of practice for welding training, recruitment and retention as well as tasks and creative activities that meaningfully connect learners' authentic interests with skills development outcomes.

In-situ mixed reality system set-up consists of three main components (A helmet integrating an augmented reality headset, a smart interactive virtual welding guide projected inside the AR headset, and a smart glove.)

Other Projects