RemoteSpark Resiliency

Mixed Reality (MR) has its roots in the gaming industry, tracing back to its predecessor, Virtual Reality (VR). Microsoft, the leading provider of MR hardware with its HoloLens, initially conceived it as an Xbox peripheral, catering to a niche market comprising just 0.01% of users.

While VR disconnects users from reality, making it ideal for gaming and entertainment, MR quickly found its niche in industrial applications. Its ability to enhance and augment the physical workspace with meaningful digital holographic content proved immensely valuable. However, despite this divergence in application, most MR applications are still developed using gaming authoring tools. This poses challenges in commercial, industrial, and critical contexts, where the demands include performance in low-bandwidth scenarios, extreme temperatures, and situations involving rapid movements, such as in tactical vehicles or ambulances.

Compounding the issue, these applications often rely on cloud services that assume reliable uptime, robust connectivity, and ample computing power. Unfortunately, the design, quality control, and testing of these applications typically focus on controlled environments within office buildings and innovation labs, neglecting real-world conditions. The question then arises: How can platform providers ensure the performance of MR solutions in any challenging environment?

At Kognitiv Spark, we prioritize the development of resilient and field-tested systems as a core aspect of every feature in our performance support tool, RemoteSpark™. Our commitment lies in creating a Mixed Reality solution that excels in the toughest and harshest environments worldwide by minimizing our reliance on bandwidth and cloud computing. Here's an overview of how we achieve this goal.

Our Approach

Foundation: SparkPlug™

Our journey toward building a truly resilient solution begins with the development of our own rendering engine, known as SparkPlug. This proprietary platform serves as the foundation for all our client applications.

While gaming applications excel in creating immersive games, we have taken a different approach by maintaining complete control over the development frameworks, creating many of them internally. This allows us to optimize the engine and tooling specifically for our target hardware, enhancing both resiliency and security. Although SparkPlug draws inspiration from gaming principles like 3D rendering and physics engines, we have streamlined its functionality to cater exclusively to industrial use cases. By fine-tuning the development foundation, we ensure optimal support for the hardware and application categories we target.

Hardware Operational Metrics

Collecting data from the operating system, network connectivity, and hardware sensors is of utmost importance. This invaluable information enables us to understand and utilize the state in which the mixed reality device operates. By leveraging this data, we can train the rendering engine and application to adapt in "stressful" environments, thus optimizing the hardware's performance when it matters most.

Network Connectivity Quality

We actively monitor key network metrics such as bandwidth, latency, and jitter. This data is then utilized by the runtime to dynamically adjust the video call's bandwidth and the amount of data transmitted between the client and the server. This adaptive approach greatly benefits the application, as it prioritizes consistent processes over excessive speed, ensuring a smoother and more reliable user experience.

Temperature

Temperature, particularly high temperatures, can significantly impact electronic devices. When devices are used outdoors, especially in direct sunlight, they are prone to overheating, leading to malfunctions and sudden shutdowns. To address this issue, we are implementing metrics within the software to mitigate the strain on the device as it approaches its maximum operating temperature, thereby maintaining cooler temperatures and prolonging its functionality. This approach may involve measures such as reducing the resolution of the video stream, decreasing the polycount in the holographic immersive view, and minimizing traffic to the server.

An essential aspect of thermal monitoring is to inform the user about the device's heat levels and the possibility of shutting down to prevent overheating. Instead of an abrupt and unexpected shutdown, we believe it is more beneficial to implement a notification system that warns users when the device is approaching its thermal threshold. This proactive approach allows users, particularly those relying on the device for complex tasks in challenging environments, to be aware of the situation, similar to the common practice of devices issuing notifications when their battery is running low.

Spatial Tracking and Movement

Addressing this aspect presents a challenge. Currently, Mixed Reality systems are often tested in controlled environments similar to VR systems, where movement is limited or even absent.

However, for MR to truly blend physical and virtual worlds, we must consider that the device won't always be in a static, well-lit environment, and user movement won't always be slow and methodical. Real-world usage of industrial MR systems is far more dynamic and intense.

What happens when users enter an elevator, swiftly navigates through narrow corridors, encounters a dark space on a ship, or operates within a fast-moving tactical vehicle or ambulance?

When an MR device loses its tracking capability, it enters a "Tracking Lost" state, akin to being blindfolded. Since the device needs to map physical boundaries (e.g., walls, floors, ceilings) to accurately position holographic content, overwhelming the sensors can lead to an inability to calculate space and position, resulting in rendering failure or undesirable behavior. In worst-case scenarios, when tracking is lost, we have a solution: video call integration.

In dynamic situations, finding the right balance between relying on digital content and having a second set of eyes to assist in successful task completion is crucial. We have designed an application that safeguards the critical elements of MR support. Even when tracking is lost, video calls remain available, enabling continuous connectivity despite challenging environmental conditions.

The Servers

When building our software, we have to consider the network connection, the temperature of the device, the battery life, and if the device understands its physical environment before rendering one pixel of a hologram – and this is just for the client application. We have not even mentioned the work put into the could infrastructure to support the MR experience.

We offer two types of backend systems: an on-premise network solution and a SaaS solution powered and protected by Microsoft Azure. In this section, we will focus on our cloud services.

When it comes to scalability, security, and leveraging cloud-first technologies to develop innovative applications, we deploy our services using cutting-edge cloud technology. This approach ensures the secure and uninterrupted operation of our system. For instance, we utilize Azure's monitoring system to track system metrics and receive alerts in case of abnormalities or issues. To enhance reliability, our solution is deployed across multiple data centers worldwide, eliminating any geographic single point of failure. Additionally, we leverage Content Delivery Networks (CDNs) to efficiently deliver binary data to end-users, utilizing servers located as close as possible to our customers.

For 3D rendering, we do not rely on cloud-based computing. This means that the client does not require a stable and high-speed internet connection to display holograms for task assistance. Our system is intelligently designed to function even in scenarios where it cannot establish a connection, ensuring that it can continue to provide the most essential functions to assist the user wearing the MR device.

____

Striking the right balance between resiliency and user experience is crucial for the success of the Industrial Metaverse. We recognize the importance of leveraging the strengths of gaming platforms in constructing immersive 3D worlds, while also making careful decisions about which flashy elements to exclude in order to maintain resiliency. Building a genuine industrial MR application is indeed a delicate balancing act, but it is precisely this ability to navigate that delicate balance that sets us apart as market leaders in this field.

What’s next? I am going to strap a mobile StarLink unit to an offroad side-by-side and head into the wilderness and see how the system holds up. Please follow these adventures on Twitter @KognitivSpark or my LinkedIn profile. ~Ryan