Overview
Vuforia Chalk, PTC’s augmented reality (AR) remote assistance platform, enables experts to visually guide technicians through complex procedures with intuitive, spatially aware annotations. The product enhances real-world collaboration, enabling organizations to rapidly and cost-effectively deliver AR to service and operations technicians using the mobile devices they already have by connecting frontline workers to global experts. In industrial operations, the context dictates the tool. A technician repairing a server rack might need the precision of a mobile screen, while a worker climbing a wind turbine needs their hands completely free for safety. Vuforia Chalk addressed this by enabling remote experts to guide technicians via live video and anchored AR annotations. My work spanned the entire product suite, including desktop, mobile, and RealWear headsets. In this case study, I show two feature developments we have implemented: (1)multi-expert flows for Chalk mobile and (2) voice-driven Interface for Chalk RealWear, which improve operational efficiency at an enterprise scale.
Research & Design
Enterprise Product Design /UX · AR / spatial interaction · Cross-platform UI/UX design · AR spatial awareness systems · Design research
- Duration: 2021
- Partners: RealWear, Microsoft HoloLens
- Team: Fas Lebbie, Dev Yamakawa
My Role
Received mentorship from my design manager at the time in contributing to the end-to-end product design process for AI-driven AR features. My focus was on defining a flexible interaction strategy that could scale across different hardware form factors, ensuring that a session felt enterprise-secure on a phone, while remaining OSHA-compliant and safe on a head-mounted display.
I contributed to UX research to understand the specific environmental constraints faced by our users. By analyzing how technicians operated in the field, I translated raw insights into interaction designs, for example, building models, identifying that mobile users were "Orchestrators" managing the crisis, while headset users were "Fixers" executing the solution.
Designed cross-platform enterprise UX and AR spatial interaction systems across desktop, mobile, and RealWear, creating cohesive and intuitive user experiences while creating divergent UI patterns, hierarchical touch controls for Mobile, and phonetic voice commands for RealWear while respecting the native behaviors of each device.
Problem Context
The Core Friction: Distance vs. Downtime Industrial teams have traditionally relied on flying experts to job sites to resolve equipment failures. This model is slow and expensive, resulting in massive operational downtime. Frontline technicians lacked real-time visual support, leading to inconsistent fixes and frequent repeat visits. Vuforia Chalk addressed this by enabling remote experts to see what the technician sees, providing step-by-step guidance through live video and anchored AR annotations. Two experience gaps that emerged as we scaled that threatened enterprise adoption:
-
The Mobile Orchestration Gap. Our original 1-to-1 calling model broke down in the real world. Industrial troubleshooting is rarely a solitary act; it is a “committee event” involving a site manager, a remote vendor, and a headquarters expert. When these groups tried to collaborate on Mobile, it was chaos. There was no hierarchy, people talked over one another, and crucially, large enterprise clients blocked adoption because they couldn’t secure the “room.” We needed to transform the app from a simple phone into a secure command center that could manage permissions, video sources, and access control.
- The RealWear “Voice Fatigue” Problem. Simultaneously, on RealWear, we faced a usability wall. While the device is voice-first, the native input experience wasn’t designed for noisy factory environments. Entering complex session codes with standard speech recognition led to high failure rates, and technicians had to repeat themselves frequently. This “Input Friction” was so high that users would often abandon the headset entirely in favor of a phone, defeating the purpose of hands-free safety. We needed to upgrade the voice UI to be deterministic and error-proof.
Chalk Mobile Support
Supporting technicians with live AR remote guidance make it possible for users to realize the potential of combining augmented reality with remote assistance.
Featured Designs
For Chalk mobile, I helped design a Multi-Expert Session Architecture that transformed the app into a secure command center. Introducing a strict Host/Participant hierarchy with Lobbies and Waiting Rooms empowered users to manage access and direct video broadcasting during complex group calls.
Hands‑Free AR Support
Chalk for RealWear increases workplace productivity and safety with hands-free remote assistance so your frontline employees can focus on getting the job done quickly and efficiently.
Featured Designs
To address input paralysis, I contributed to the design of a voice-first user experience to reduce voice fatigue. We overhauled the standard input interactions with a phonetic soft keyboard (e.g., “Alpha, Bravo”). This replaced error-prone natural speech with precise, military-grade commands that work in high-decibel environments. Additionally, we flattened nested menus into high-contrast tiles, ensuring that every visible function was an instant, pronounceable command.
AR interfaces empower over 1M technicians with guidance, improving first-time fix rates and reducing service calls.
Operational Improvement (Mobile)
By enabling Multi-Expert Sessions, we ensured the right experts were in the room on the first call, directly improving operational outcomes (reducing repeat visits).
Efficiency Gain (RealWear)
By replacing error-prone manual entry with the Phonetic Voice UI, we reduced the time it takes a technician to join a session by 40%, eliminating the “fumble time” at the start of a crisis.
Reflections & Impact
For the multi-expert session feature, the introduction of Waiting Rooms and Host Controls satisfied critical SecOps requirements, unblocking deployment in high-security environments that previously rejected the tool. We satisfied the strict security and hierarchy requirements of enterprise IT, removing the primary blocker to large-scale deployment. Additionally, by designing a Native Voice Interface for RealWear, we removed the physical barrier to entry. These feature cases are among many in my time at PTC that helped improve my approach and experience in cross-platform design, where designers must be immersed in the distinct “physics” of each platform (Touch for Control, Voice for Action) to create a user experience that solves real user problems.
