Report: Your Brain on Augmented Reality

Augmented Reality in Manufacturing, It’s All in the Neuroscience

Augmented reality (AR) refers to a class of tools and technologies that overlay virtual objects and information onto the user’s view of the physical world that become a part of the real-world. AR is taking the world by storm. From the explosion of interest in games like Pokemon Go in 2016, to the growing number of use cases we see today, with each passing day, new AR tools and capabilities are being introduced.

Although the term “enterprise-grade AR” is common, no objective and rigorous standards have been developed to define AR solutions along a continuum of excellence Click To Tweet

Although the term “enterprise-grade AR” is common, no objective and rigorous standards have been developed to define AR solutions along a continuum of excellence. This report represents a first step toward developing an objective, rigorous standard for defining enterprise-grade AR that is grounded in the neuroscience of learning and performance. Regardless of the application, it is the same brain that learns and drives performance. Thus, a deep understanding of “Your Brain on Augmented Reality” is the key to building the highest quality enterprise-grade AR solution.

This is an insightful quote from Albert Einstein that is supported by the neuroscience of learning and performance. But why? What is it about experience that is so rich that it is fundamental to learning, and why is information so much less effective?

The Neuroscience of Learning and Performance
“Learning is an experience. Everything else is just information”  Albert Einstein

The human brain is comprised of at least three distinct learning and performance systems (see Figure). As Einstein so eloquently stated, experience is at the heart of learning. Importantly, it is also at the foundation of enterprise-grade AR. AR provides augmented information co-located on the users’ current reality or experience.

The experiential system has evolved to represent the sensory aspects of an experience, whether visual, auditory, tactile or olfactory. Every experience is unique, adds rich context to learning and is immersive. The critical brain regions associated with experiential learning are the occipital lobes (sight), temporal lobes (sound), and parietal lobes (touch).

The cognitive system is the information system. It processes and stores knowledge and facts. Cognitive information comes in the form of text, graphics, sound, or video and is limited by the learner’s working memory and attention span. Critically, these are limited resources and often form the bottleneck that slows learning with more information coming in and available to the learner (the green arrows) than can be processed (the red arrow). This system encompasses the prefrontal cortex and hippocampus.

The cognitive system is the information system. It processes and stores knowledge and facts. Information comes in the form of text, graphics, sound, or video and is limited by the learner’s working memory and attention span Click To Tweet

The behavioral system in the brain has evolved to learn motor skills. The critical brain structure is the striatum. It is one thing to know what to do, but it is completely different (and mediated by different systems in the brain) to know how to do it. Processing in this system is optimized when behavior is interactive and is followed in real-time (literally within milliseconds) by corrective feedback. Behaviors that are rewarded lead to dopamine release that incrementally increases the likelihood of eliciting that behavior again in the same context. Behaviors that are punished do not lead to dopamine release thus incrementally decreasing the likelihood of eliciting that behavior again in the same context. Real-time feedback is critical because the brain’s response fades quickly (within a few 100 milliseconds), meaning that learning and unlearning will not occur.

A Neuroscience-Based Definition of Enterprise-Grade AR

Now that we understand the neuroscience of learning and performance, we can objectively and rigorously define the neuroscience-based enterprise-grade AR standard, and identify the AR capabilities needed to meet this standard.

An enterprise-grade AR tool must engage experiential, cognitive and behavioral learning and performance systems in the brain in synchrony Click To Tweet

– First, and foremost, an enterprise-grade AR tool must engage experiential, cognitive and behavioral learning and performance systems in the brain in synchrony. This can be achieved by engaging the experiential learning system with AR assets overlaid on the real-world that provide information to the cognitive learning system and real-time interactive feedback to the behavioral skills learning system. This builds a cognitive understanding while simultaneously developing a strong behavioral repertoire, all within a real-world setting.

– Second, the AR assets must be presented to the user in a way that minimizes the load on the cognitive system. Recall that the bottleneck in the cognitive system is working memory and attention that are both limited capacity resources. Regardless of whether the AR asset is text, graphics, sound, video, or holographic, it must be presented in a way that minimizes the cognitive load on the user by presenting the user with what they need, where they need it, when they need it. This is not a trivial problem to solve and the type of asset, as well as the timing and location of the asset will be determined by the use case, but the goal must always be to reduce the load on cognitive processing.

– Third, the AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time. AR assets that guide users’ behavior must be temporally linked to the users’ behavior. In other words, as a user completes a step, AR assets that guide the next step must be presented in real-time. In addition, these assets must react to the specific behavior elicited by the user. If a correct behavior is elicited by the user, then the next asset in the chain should be presented. On the other hand, if an incorrect behavior is elicited by the user, the AR solution must provide some form of real-time corrective feedback to the user. This is where a combination of AR and the Internet of Things (IoT) would be advantageous. Information from IoT will be shared with the AR solution in real-time that will drive the spatial and temporal presentation of AR assets. This real-time interactivity will effectively engage the behavioral learning system and quickly build a strong behavioral repertoire.

Any AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time Click To Tweet

Enterprise-Grade AR in Manufacturing – A Use Case

The Problem: A major threat to the manufacturing and industrial sector is the increasing workforce skills gap. Research from the Manufacturing Institute suggests that nearly 10 million manufacturing jobs will be needed in the next decade with millions of these going unfilled. To make matters worse, new workers will not have the expertise of seasoned veterans, leaving the manufacturing sector with a less productive, and potentially more error-prone, workforce.

Research from the Manufacturing Institute suggests that nearly 10 million manufacturing jobs will be needed in the next decade with millions of these going unfilled Click To Tweet

What is needed are methods for training the new workforce that quickly and efficiently instill the subject matter expertise and behavioral skills needed on the factory floor. For example, when training a new worker to operate and maintain a piece of equipment, the ideal approach is to train the vocabulary to describe and label each part while simultaneously training the behavioral repertoire needed to effectively operate and repair the equipment. This coordinated training approach reduces the cognitive load on the user while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice. Taken together this speeds learning and retention, quickly builds behavioral mastery, and streamlines performance. Ultimately this will lead to fewer errors and reduced time to completion by minimizing unscheduled downtime.

The Solution: Consider the problem of equipment operation and maintenance. In this case, a worker might don a Microsoft Hololens or some other hands-free AR device. Let’s examine the capabilities of this solution to determine whether it meets the enterprise-grade standards outlined above.

The AR assets must be presented to the user in a way that minimizes the load on the cognitive system Click To Tweet

1. An enterprise-grade AR tool must engage the cognitive, behavioral and experiential learning systems in the brain in synchrony. With the Hololens (or any other hands-free AR device) cognitive information can be provided in the form of a virtual dashboard that includes numeric data and step-by-step instructions that guide the worker’s behavior. The AR solution is wearable and hands-free, thus allowing the worker full range of motion to complete behavioral tasks, and to build a behavioral repertoire in a naturalistic setting.

2. The AR assets must be presented to the user in a way that minimizes the load on the cognitive system. Recall that the critical factor here is to present the user with what they need, where they need it, when they need it. What asset is needed (whether text, graphics, sound, video, or hologram) will depend upon the specific use case. In a typical manufacturing setting text and graphics are the dominant AR assets. Where assets need to be presented to minimize cognitive load is with a co-located, contextualized overlay so that the worker never has to shift gaze or attention away from the task at hand. Every gaze or attention switch slows the process and increases the likelihood of error. Finally, when the asset is needed is just-in-time. As soon as the asset is needed it must be presented to the worker. Every second delay, or need to search long-term memory increases the cognitive load slowing the process and increasing the likelihood of an error.

3. The AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time. Let’s assume that the information in the virtual dashboard is providing the worker with step-by-step guidance while monitoring the correctness of the workers’ actions. Most likely an IoT solution is working in tandem with the AR system to determine what assets to present and to determine when and where they should be presented. Once a step is completed successfully, the next instruction is presented immediately because the IoT solution knows in real-time that the step was completed successfully. If a step is not completed successfully the worker is immediately presented with corrective feedback and a chance to complete the step successfully, again because the IoT solution knows in real-time that the step was not completed successfully.

This broad-based and coordinated brain activation that results from enterprise-grade AR solutions reduces the cognitive load while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice Click To Tweet

With an enterprise-grade AR training tool, the training can be repeated as many times as the user would like, and the user can be placed in a broad range of situations. For example, the user can be tested by placing them under time pressure, or they can receive additional training on rare, but potentially dangerous, situations.

This broad-based and coordinated brain activation that results from enterprise-grade AR solutions reduces the cognitive load on the user while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice. Taken together this speeds learning and retention quickly builds behavioral mastery and streamlines performance. Ultimately this will lead to fewer errors and reduced time to completion by minimizing unscheduled downtime.

Recommendations and Closing Remarks:

Whether you are an AR developer or user, it is important to understand which AR solution to use when. The best way to make this determination is to understand “your brain on augmented reality” so that you can select an enterprise-grade AR solution.

Here are some guidelines:

– If the users of your AR solution need to learn critical motor skills along with relevant subject matter, as in healthcare or manufacturing, then an enterprise-grade AR solution that uses a wearable device to keep the hands free to work in recommended. This AR tool should present AR assets, such as text, graphics, sound, video, or holograms, in a co-located fashion where the user needs them, and just-in-time when the user needs them. This will reduce the cognitive load on the user that is an ever-present bottleneck on cognitive processing. These assets must be interactive and temporally linked to the users’ behavior. Most likely this means that an industrial IoT solution is being combined with the AR solution to achieve real-time interactivity. If a correct behavior is elicited by the user, then the next asset in the chain should be presented. On the other hand, if an incorrect behavior is elicited by the user, the AR solution must provide some form of real-time corrective feedback to the user.

This AR tool should present AR assets, such as text, graphics, sound, video, or holograms, in a co-located fashion where the user needs them, and just-in-time when the user needs them Click To Tweet

– If you decide to choose a non-interactive AR solution, consider the use case as you decide between a tablet-based solution and a hands-free wearable solution. If your goal is behavior skills development, then choose a wearable, hands-free device. Allowing workers to generate natural motor movements, instead of holding the tablet with one hand while performing some task with the other, will go a long way toward building muscle memory.

– The specific use case will determine whether location-based assets such as GPS, beacon, or triangulation tools are needed. Similarly, the range of content types available should be determined from the use case. For example, holograms, generated voice and graphics may be critical when a 3-d mental representation of a complex piece of machinery, system in the body, or retail setting is needed to build a cognitive and experiential understanding. The more that the user needs continuous eye fixation on the task at hand, the more you want graphic overlays (perhaps in the form of 3-d holograms) and generated voices. Keep in mind what the brain needs, not what is flashy.

– In this report, we focus on training as the use case, but what if you want an AR tool to enhance performance of a repetitive task such as navigating an automobile or managing inventory? In these cases, the goal is to streamline the performance of a well-understood process, not to impart a detailed cognitive and behavioral understanding. When performance is the goal, the key is to reduce cognitive load by reducing working memory demand and the number of attention shifts. An AR tool that automatically detected the user’s location and directs their focus is ideal. In navigating an automobile, the ideal AR solution would overlay a virtual map on the windshield offering turn by turn instructions through non-distracting video and audio. In the case of inventory management, the ideal AR solution would automate the counting of items, facilitate registering the information, flag cases in which more inventory must be ordered, then direct the user to the next item. In both cases, this significantly reduces the cognitive load which reduces errors and speeds time to completion.

In Summary:

AR tools have enormous potential in manufacturing and many other sectors including automotive, medical, and retail, to name a few. Although the necessary AR features depend upon the use case the goal is always to reduce cognitive load by broadly engaging experiential, cognitive and behavioral learning centers in the brain. Enterprise-grade AR tools have the potential to engage the learners’ brain with co-located information, just-in-time experiences and behavioral training that provides the users’ brain with what they need, where they need it, and when they need it. This speeds time to productivity while reducing errors and costs.

AR tools have enormous potential in manufacturing and many other sectors including automotive, medical, and retail, to name a few Click To Tweet