Remote Assistant Object Detection AR/VR

Spare Parts Recognition for Hands-Free AR Interface for Remote Assistance Industrial Work

Project Overview

At PTC, I partnered with the innovation team to launch a Real-Time Object Detection (RTOD) system, transforming how service technicians locate and manage spare parts. By leveraging artificial intelligence and augmented reality, factory workers can instantly identify any part of the equipment simply by taking a photo, helping manufacturing companies eliminate the time-consuming process of manually searching through parts catalogs. Through integration with Windchill database systems and deployment across multiple platforms (mobile, Hololens, and RealWear), our work pushed the boundaries of visual recognition technology, delivering an efficient solution that addresses the $2 billion service spare part catalog industry challenge.

My Role

As a Senior Product Designer at Vuforia, working within the Innovation Runway Team, I led experience design across multiple AR platforms. I also assisted with qualitative and quantitative research methods, developed intuitive XR interfaces for AR devices, websites, and mobiles, and deployed AR Remote assistance solutions.

Confidentiality: The design process shared reflects my perspective, not PTC's. Specific details have been modified to showcase my design approach.

Project Duration: February 2021- August 2022

Key Partners: PTC, Vuforia, Azure, TensorFlow

Team: Fas Lebbie, Dr. Eva Agapaki, PTC Innovation Runway Team

Problem Context

The current spare parts identification system in manufacturing creates significant inefficiency, with service technicians and factory workers wasting hours searching through outdated catalogs to locate necessary tools and components. This manual process, particularly prevalent in complex machinery environments like the automotive industry, leads to frequent inventory mismanagement. Technicians relying on traditional part books often end up ordering incorrect components or duplicating existing inventory when making online purchases. These problems stem from an identification framework that hasn’t evolved with modern technology, resulting in time-consuming processes and requiring documentation in the worker’s primary language to be effective.

Design Interventions

We developed the “AR Hand Book” app, enabling users (automotive industry service technicians and factory operators) to take photos of any part. The system instantly identifies the component through our AI and machine learning framework and provides all relevant data from the Windchill database (including part numbers, CAD models, drawings, and BOM). The platform achieved successful validation from in-house engineers and customers, with major businesses adopting the technology, including Porsche for mobile quality inspection and Liebherr for product inspection.

Workshop pegboard with numbered tool organization system for quick identification in industrial maintenance settings.

Mobile interface showing camera view and main menu with Classify/Annotate/Detect options for part recognition.

Project creation screens for setting up new part identification tasks in the AR Handbook app.

Object annotation process showing selection, bounding box creation, and labeling of industrial measurement device.

Training interface with deployment options and main menu for selecting part recognition functionality.

Model selection interface for loading predefined component recognition models from Windchill database.

Tool detection sequence showing application loading, project initialization and camera-based identification of workshop scissors.

Real-time tool recognition with confidence metrics displayed through minimum probability sliders for scissors identification.

Multiple tool identification screens showing automated recognition of wrenches, measuring tape and scissors.

Parts management screens showing identified component with color variations from PLM database integration.

Complete user flow overview showing all pathways through the AR Handbook app's three main functions.

Scroll to next section

My Approach

We approached this challenge by starting with a deep understanding of the parts management ecosystem. I researched current manual processes, competitive landscape, and industry verticals for technological solutions, proposing hypotheses to test. This structured approach enabled thorough investigation through a research-first methodology that shaped our design approach. We analyzed Vuforia’s internal strengths to ensure project feasibility, leveraging PTC’s existing CAD infrastructure and training content to establish a reasonable universe of objects (narrowing potential parts to 20-200). We could ensure practical implementation by utilizing AR technology that is feasible for ubiquitous smartphone technology and building upon a base of 30,000 customers who use CAD and work with 3D imaging. This strategy guided our design process—from leveraging AI training models to implementing intuitive AR interfaces across platforms. Each feature was built to enhance productivity while simplifying parts identification across user contexts and device environments.

Design Process

Design Research & Strategy

The research explored four key areas: current solutions, competitive landscape, technological verticals, and hypothesis formulation. We first documented the manual process used by service technicians, who rely on old part books to identify components for repair before purchasing from online retailers. This process is time-consuming, risks ordering the wrong parts, and requires manuals in the worker’s primary language. We examined the global visual search market (valued at $10 billion in 2018 and projected to reach $28-60 billion by 2027), identifying $2 billion related explicitly to service technicians’ and factory workers’ needs. Key players like ASOS, Ted Baker, and eBay were leveraging similar retail technologies. ASOS’s style match allows purchases based on outfit images, Ted Baker launched shoppable videos, and eBay implemented visual search technology. Our analysis of technological verticals revealed that the automotive industry had the greatest potential due to the need to recognize individual parts within complex machines. We identified three primary end-user groups: factory operators, service technicians, and customers performing their own maintenance. Medium-sized manufacturers emerged as our target market due to their quantity (approximately 10,000 worldwide), high revenues ($400M-$10B), and desire for efficiency improvements. Understanding these pain points and market opportunities established our baseline for creating a solution to digitize and streamline this process through visual recognition technology. This research guided our technical approach, leading to a three-stage process: Classify (uploading and defining client’s parts), Annotate (associating customer-specific names and data with parts), and Detect (reliably identifying parts through AI).

Summary of Findings

Our investigation revealed that PTC/Vuforia should intervene in this industry for several key reasons: PTC already had the capacity to provide end-to-end ‘Create-Manage-Deliver’ service information on spare parts due to its established CAD infrastructure spanning authoring, management, and delivery of all needed content types. PTC already possessed training content that could help the system establish a reasonable universe of objects and narrow down potential parts to 20-200. Additionally, AR technology was feasible for ubiquitous smartphone technology held by service technicians, and PTC had a customer base of 30,000 who regularly worked with CAD and 3D imaging, providing an immediately accessible market. Through our user research, we identified four specific personas with distinct jobs to be done. These insights directly shaped our product requirements and user experience design:

Jim Lee (Service Operator/Technician)
- “Today, I’m using a part catalog (physical book or online) to find the part I’m looking for. Many times, I make mistakes, and the process takes time.”
- Needs help to avoid making mistakes when ordering parts
Hannah Jamser (AR Service Provider)
- “Today, I provide AR apps to manufacturing units, but there is a limited set of 3D models users can automatically place in the physical world.”
- Serves as the intermediary between manufacturers and software companies
Alez Rodriguez (Factory Operator)
- “I end up ordering obsolete parts that I don’t need and have increased operational costs.”
- Responsible for optimizing the processes in the factory
James Romero (CAD Author)
- “It’s hard to organize my database so that CAD models can be used by AI algorithms directly.”
- Organizes and maintains CAD database

Prototyping & Implementation Strategy

Our prototyping process emphasized developing a seamless user experience across multiple platforms while building a robust AI recognition system. We created a detailed user flow sequence that included both the annotation and training phases:

Annotation Flow (Steps 1-3):

Take a screenshot with a mobile app that has a class 1 object (e.g., screwdriver)
Annotate in the mobile app (draw bounding box)
Send annotated images to the Azure web server

Training Flow (Steps 4-8): 4. Train in Azure Custom Vision server (advanced training may take hours) 5. Export trained model to iOS/Android 6. Load the trained model in app project 7. Predict class in an image taken from the mobile app in real-time 8. The user accepts, modifies, or rejects bounding box prediction

We developed an AI training model for the backend utilizing tools like Onshape, Blender, and ParaSOLID to generate synthetic images and point clouds, which fed into a vanilla AI network for classification. Instead of using traditional annotation methods, we developed the RTO2-3D network, which was trained on both images and point cloud data. For the front-end user experience, I designed high-fidelity interfaces for mobile devices, tablets, and RealWear AR headsets, adhering to platform-specific best practices. For mobile, we focused on 2D interactions, allowing users to act as a “magic hand,” making inputs into the semi-real world. For tablets, we incorporated placeable pins on CAD models to help users make annotations, ensuring clickable functions were within reach. For RealWear AR, we limited background functions to preserve battery life. Our implementation strategy was structured with clear execution parameters:

Time:

6 months for gradual adoption with Vuforia Studio widgets
1 year for RTOD services suite
Across four sprint phases

Budget:

Approximately $1.7M

Human Resources:

7-10 FTE

Working within these parameters, we successfully delivered a product with four unique characteristics:

Efficient: Instantaneous predictions with constant improvement
Versatile: Adaptable to customer environments
Persistent: Constant predictions around the 3D space

Cross-platform: Integration between Unity Engine and Vuforia Studio

Summary of Findings

Scroll to next section

Reflections & Impact

Within the first year, we achieved four key milestones that validated both our approach and the market need 1) Disrupted the $2B service part catalog business by creating a digital solution that eliminated the need for physical catalogs and manual searches 2) Developed a new method for CAD model classification that enables automated mappings in PLM systems 3) Created RTOD web-services that support features essential for part classification and recognition 4) Secured customer and internal validation from both in-house engineers and external clients. This innovation quickly gained traction in the industry, with four major businesses adopting our RTOD system: Porsche implemented it for mobile quality inspection, Liebherr for product inspection, PUIG for part recognition, and IKEA included similar technology within their IKEA Place app. The project’s impact extended beyond these initial implementations, creating significant workflow improvements: technicians eliminated time wasted searching through catalogs, the system enabled part identification in 10 languages, and reduced ordering mistakes. Our solution’s long-term transformative potential comes from four unique characteristics:

Efficient: The system makes instantaneous predictions that constantly improve over time through machine learning, becoming more accurate with each use.
Versatile: The solution adapts to any customer environment, developing based on their specific needs and feedback
Persistent: The system makes constant predictions throughout the 3D space it operates in, ensuring reliable part recognition regardless of viewing angles and conditions.
Cross-platform: The integration between Unity Engine and Vuforia Studio makes the technology accessible across multiple platforms and devices

By leveraging these advantages, we successfully disrupted a $2 billion market that had long relied on traditional service part catalogs, creating a backend infrastructure that now supports essential features for part classification in PLM systems and recognition in client applications.

Previous Project

All Projects

Next Project