Expanding the Reach of Mixed Reality With RemoteSpark
When we started on our journey to create a system that would help frontline workers, we knew the information that the workers needed could come from all sorts of sources, both human and machine. We then had to determine which kind of data and in what forms was the most useful to the frontline worker.
The Journey
The First Step - RemoteSpark 1.0
The focus for 1.0 was on the frontline workers that needed troubleshooting assistance. The worker requiring help would connect a mixed reality video call to an expert for guidance on a complex task or troubleshooting issue. The expert could then see a first-person video feed of the situation and push 2D and 3D content to the worker to augment their physical space with digital assets to assist them.
A Big Stride - RemoteSpark 2.0
We expanded the reach of mixed reality beyond being solely a tool for remote assistance, we empowered the worker whether or not an expert is connected.
We empowered the worker by:
1. Digital Assets in the Physical Space
With RemoteSpark 2.0 workers can place 3D animations (troubleshooting or procedural guides), PDFs, or photos in their physical space. Anyone authorized from your organization that puts on a mixed reality device like a HoloLens or Magic Leap device can see the digital assets associated with that location. Imagine being new to a job, putting on a pair of mixed reality glasses and there is everything you need to know about the location in the location!
2. Palm Panel
If the content you need is not in the room, then we have a holographic menu system that you can open and load 3D mentoring files, PDF instructions, diagrams, or instructions to assist you.
3. Mixed Reality Support Call
If there is no content in the room or in the menu that can help you, a connection to an expert with a video call can be established.
Is there data in the room to help you? Is there data in the system to help you? Is there a person I can connect to for help?
RemoteSpark 2.0 has enabled workers to have layers of assistance that enable autonomy, while also having support from experts if required.
RemoteSpark 2.1
Is there data needed by a worker that is not a common data type like a JPEG, PDF, or 3D animation?
The next version of RemoteSpark coming in Q1 2024 will include displaying machine-generated data, not just files from a person, on a server, or in the cloud. This creates the opportunity to create dynamic data tailored to the situation instead of data that has been curated and prepackaged. We are making the leap from preparing a collection of data you might need, to data coming from the environment in near real-time.
We categorize the dynamic data into four buckets:
1. Remote Spark Dashboard API - Display of Sensor and IoT Data
Anyone who has read an industrial magazine or website lately has seen the discussion about industrial digital transformation and IoT being a central message of this transformation.
What data does the worker need from their physical environment provided by IoT or sensors to do their job?
Our focus is to create holographic screens to display important data from devices that have no screens. This way as the worker adjusts or repairs a system, they can see important metrics holographically without having to put down their tools or take off their gloves and grab a tablet to monitor the situation.
3GA Marine, a naval architecture engineering firm in Canada, is piloting our Dashboard API to use it to display operational metrics holographically to the technician as they perform the ship maintenance. This allows the technicians to see in near real-time the effects of the tasks plus a mixed reality call can be established via Star Link so the process can be monitored by land-based experts while the maintenance is done at sea.
The video below was captured with Magic Leap Spectator, a soon to be released application that allows your phone to capture digital and real-world content.
Now not only do you have holographic instructions from 3D animations or PDF diagrams, but also live dashboards showing the important data points to further enhance your maintenance and troubleshooting procedures.
2. RemoteSpark Vision API – Inspection and AI Integrations
Customers have a diverse range of inspection systems. How do you handle the integration?
Instead of integrating bespoke systems, we created a system by which you add a custom voice command to trigger a videoframe (a photo) from the mixed reality device camera, the captured frame is then sent securely to an external service endpoint for processing. Sounds techie, eh?
For example, an organization that does inspections may want to verify the task is complete using their own AI or vision inspection system. We add a voice command “RemoteSpark Verify Inspection” for that organization. That voice command triggers a photo and associated metadata from the MR device to be sent to their cloud application for processing.
Once the data is processed by the customer’s application, it has the option to send a command back to RemoteSpark. These commands are mainly to instruct RemoteSpark to show content that needs to be displayed in result of the photo processing. For example, the worker says, “Inspect This”. A photo is sent to their company’s verification AI and the AI notices issues. The AI then draws boxes around the issues in the photo and the modified image is displayed for the worker as an image or PDF. Instant feedback. Think about that, the worker does not have to put down their tools, take off their gloves and grab a photo with a phone/tablet, and then wait for the results. Now as they work on the task, they can get continuous feedback until the job is done right.
3. Content Publish API – Push Content to RemoteSpark
Enterprises have content management systems. RemoteSpark has an Organization Library with the digital assets curated to help workers holographically. The Content Publish API was designed so integrations can be made to assist automations to publish data from the enterprise to RemoteSpark.
There are two main use cases for this API.
1. Publishing from a Content Management System (CMS). If you have a system that manages documents, workflows can be added to publish to RemoteSpark when needed.
2. Ad-Hoc Publishing. Use case example, a ticketing system. A customer has a ticketing system that could assign a ticket to a worker, notify the worker of the work order (via corporate email or text), and then publish the required documents to RemoteSpark so that when they get on-site, they have the data they need. The data could be PDF, JPG, 3D Files, and even an IoT Dashboard.
4. Telemetry API – You can’t manage what you can’t monitor.
We have a web-based dashboard coming soon which customers can log into in order to monitor their RemoteSpark usage. As a customer scales, they have requirements unique to their operations. We are providing an API that enables an organization to download their data feed which they can incorporate into their own workflows, dashboards, and even their AI for sophisticated use cases.
In Closing
RemoteSpark is starting the journey to expand the reach of the data it can consume into the platform and extend the integration points that can be used for a wide range of enhancements. With these new connection points between systems, we are looking forward to the use cases you discover to further empower your operations and front-line workers.
