From Myths to Principles: Navigating Instructional Design in Immersive Environments Part 5 Myth: Learners learn more
The
foundation of all learning, or child’s first book, by which a child
will learn more in one month than by many others in twelve, Author Unknown, Date 1800. Source: Compositor, University of Birmingham
Myth: Learners learn more in immersive experiences
This myth shrouds itself within a cloak of research. Citations will state that learning in immersive experiences is somehow greater when pitted against an implied traditional learning approach. The claim could appear as retention, but it is related to how well the learning was accomplished when measured up against learning objectives or a final goal.
When referring to the PricewaterhouseCoopers (PwC) report results mentioned earlier in this article series, Scott Likens claimed, “We found the realism and performance feedback in virtual reality simulations helped people learn faster and retain more information around soft skills,” (Zielinski, 2021, para. 9). He claimed they retained more information. This specific claim has been repeated in academic literature, which stated, “Studies have found that students who use XR training are more engaged with the content, display more confidence with the material, and retain more information than students who use traditional training methods.” (Rode, 2024, para 2.)
A close examination of the PwC report, however, reveals that the claim was unsupported within the report’s own data. When comparing information retention in VR versus an e-learning course, the authors “quickly discovered retention scores were inconclusive, as the delta between pre-and post-assessments in each modality was not significant” (Eckert & Mower, 2020, p. 44). Thus, there was no statistical difference between VR-based, e-learning, and traditional classroom learning.
Claiming
something happened but your instrument didn’t pick it up is the knomes-did-it territory of cause-and-effect, dudes. Watch out.
The report therefore does not provide statistical evidence of more or greater learning within VR, yet it has been cited in academic publications (O’Dwyer, 2021, Etienne et al., 2022, Jelki et al., 2022; Bäckelin, 2023; Etienne et al., 2023; Lønne et al,. 2023) and touted in media outlets (Murad, 2023; Schwantes, 2020). For the dubious claims, the report has been debunked as untrustworthy (Neelen & Kirschner, 2020).
There are similar claims about greater learning retained from immersive experiences. Advocates for digital twin campus environments claimed that they “create greater retention of the information that is learned” (D’Agustino, 2022, para. 5) and “students’ grades go up” (Victory XR, 2024, Who We Are).
While this is just one tiny sentence, keep in mind how much money VictoryXR makes from these claims.
In another example, the CEO of the Miami Children’s Health System touted that learners had 80% retention after one year after using VR, but traditional learners had 20% retention. A close look at the supporting documentation shows that the CEO actually said that the difference between VR learners and traditional learners can be the 80% to 20% difference.
However, the CEO’s statement did not refer to any published results; it was opinion. The CEO explained their claim by saying that, “The level of understanding through VR is great because humans are primarily visual, and VR is a visual format” (Gaudiosi, 2021, para. 4). Cue learning styles!
Nonetheless, the quote of 80% retention has made its way into academic research (Iacono & Vercelli, 2019; Mathew & Pillai, 2020; Ternès, 2018). Some claims are extreme. One keynote speaker, Alvin Graylin, speaking as a leader at HTC (a VR headset maker) declared that as a result of VR use in the classroom, “Every single child has the potential to be a genius” (Educators in VR, 2020, 23:33).
“Every single child has the potential to be a genius.” You just need to buy a (HTC) headset.
Justification for the greater learning or retention claims seems to be conjecture. Claims refer to how real an immersive experience feels to a learner. Returning to the PwC report, Likens credited “the realism and performance feedback…helped people..retain more information.” (Zielinski, 2021, para. 9). But long term studies measuring retention are hard to find within the body of academic literature. Some studies measure retention three weeks after the immersive experiences. Given that many of the examples of immersive experiences relate to hands-on disciplines like nursing or construction, things learned in immersive experiences would be needed on the job more than three weeks after the training.
Hamilton et al. (2020) stated that finding “learning outcomes, intervention characteristics, and assessment measures associated with immersive virtual reality has been sparse” (p. 1). Beck, Morgado, and O’Shea (2023) pointed out that details of methods are missing so that outcomes become questionable, “Very few literature reviews focus on the educational practices and strategies used in immersive learning environments. Thus, the problem is that we are evaluating outcomes without a comparable way to describe the educational approaches that led to those outcomes” (p. 2). Lawson et al. (2024) completed a systematic review of immersive experiences and found that research studies rarely isolate instructional methods and conditions when describing research studies and thus impact real world classroom decisions.
Meta analyses are starting to illuminate this area. Akgün and Atıcı (2022) observed that there was only a moderate effect on learner achievement after surveying 31 studies. Kaplan, Cruit, Endsley, Beers, Sawyer and Hancock found that “XR does not express a different outcome than training in a non-simulated, control environment. It is equally effective at enhancing performance” (2020, p. 1) Some researchers are starting to incorporate machine learning and artificial intelligence into this challenge in order to determine what the published records states about immersive experiences. Markowitz et al. (2024) recently surveyed 196,734 paper abstracts with this method.
Basically
this graph says that because the numbers are so small and so close to
zero, there is no discernible effect of VR on learning, regardless of
immersion (Kaplan et al., 2020).
From Myths to Principles: Navigating Instructional Design in Immersive Environments Part 4 Myth: Learners learn faster
Credit: Burst and Canva
Dispelling Myths
With some background established on boom and bust cycles in the hype for immersive experiences (Parts 1, 2, and 3), we need to dismiss the rather rampant myths about learning within immersive experiences. In the hype, learning advantages have been overstated and over simplified. Web pages post outrageous claims (and I’ll show you!). Keynote presentations banty incredible promises (yup, it was recorded). This series addresses the four primary myths about learning within immersive experiences: that is, that it is faster, greater, active, and induces empathy.
In this Part 4, I’ll address the “learners learn faster in immersive experiences” myth. For those of you that follow my writing, you’ll know that this is Round 3 of me taking on this myth. My argument has not changed; remember this article series is an update, but not every point needs updating. However, I continue to communicate about this because the “VR learning is faster” myth continues to circulate– mostly in the reference to “4 times faster” and the PwC report. So, TLDR, the VR experience was designed to be 29 minutes long. That’s it. No longer. The classroom equivalent in content experience was designed to be 2 hours long. That’s it. 29 minutes is ¼ of 120 minutes. Someone inverted ¼ to 4x (which is factually true) and PwC who appears to have had a cozy contract with Oculus/Meta at the time, went out to trumpet the ‘four times’ from the rooftops. But students do NOT learn faster. They experienced a learning event that was designed to be faster. Had the learners spent 120 minutes in the headset, someone would have probably greedily snatched the headset off their heads and told them that they overstayed their welcome (and wondered what they were doing for the extra 91 minutes).
Myth: Learners learn faster in immersive experiences
The first myth asserts that learners learn faster with immersive experiences. Particularly, the phrase “four times faster” has taken root in the publications and in public discourse. A google search on the phrase “VR is 4 times faster” returns a plethora of results repeating the myth. 4x in the wild. And it’s not hard to catch, yo.
The source of this phrase is suggested to be one non-peer reviewed industry report by PricewaterhouseCoopers. Within the report, VR-based learning was “4x faster than classroom training on average” (Eckert & Mower, 2020, p. 8). The results of this report were then repeated in academic literature.
Do not make pie graphs that do not add up to one whole thing.
Referring to the same report, D. Clark (an educational researcher not known for getting data wrong, but he did) wrote enthusiastically that “VR was x4 faster than classroom and x1.5 faster than e-learning” (2021, p. 190). Claims that learning is completed faster attempt to represent immersive experiences as a more efficient learning method, i.e., less time to learn equals learning faster.
Tracking down how many academic papers have cited the PwC report is difficult. I’ve seen numbers as little as 4 citations and much higher numbers if I start flexing my search. Part of the problem is that folks have not cited the report (even though it calls itself a study) correctly. Some credit PwC, a few find the Eckert and Mower authors, but in general the hand wave approach to referring to the 4x data is very prevalent.
The cause of this supposed faster learning was attributed to how a VR headset isolates the learner’s perception, so that the learner is focused on the learning task at hand. In other words, less distraction equals more focus. In the PricewaterhouseCoopers report, Likens seemed to hypothesize that, “A lot of courses that normally take an hour could be completed in 20 minutes through VR because people are so immersed in scenarios, there are fewer distractions and the learning is very concentrated” (Zielinski, 2021, para. 10).
To be clear, in the PwC case, classroom learning which covered the same content was designed to take two hours to complete. The immersive experience was designed to take 29 minutes. Given that 29 minutes is approximately one-quarter of two hours, PwC inverted the time ratio and touted the line that the immersive learning was four times faster. The problem is that it is not true that learning in the immersive experience was faster. The VR-based learning took less time because it was designed to be a 29-minute one-on-one learning experience designed for a shorter total time duration. When compared to classroom learning, it is already known that one-on-one personalized learning is generally faster; it moves at the speed of the learner, not at the speed of the class. Perhaps, this is how myths begin. A kernel of truth gets extended to something with no context. Lack of context is a noted and rising problem in educational research (Williamson, 2024).
Learning faster can be confused with greater efficiency. Efficiency could have a wide range of meanings beyond just taking less time. It could also mean wiser use of resources or less teaching burden on the instructor. Another example of the loose wordplay is on an industry webpage that displayed that VR training was 50% faster than a traditional in-person medical simulation. Not stated in the distilled summary of that study is that learners scored worse in the VR training than the traditional in-person medical simulation (Katz et al., 2020).
If faster equals worse performance, this might not be the efficiency that educators are looking for.
The myth that learning happens faster continues when educators fail to acknowledge that a different instructional method was being used. When supporting using virtual reality for chemistry studies, Muhsinah Morris, a chemistry professor and metaverse program director at Morehouse said “You can’t see molecules, but in my virtual reality classroom where I taught advanced inorganic chemistry, you can. You can actually build three-dimensional representations of molecules … The learning tends to happen faster. They go on to the real situation faster.” (D’Agostino, 2022, para. 5).
Side point: Learning Chemistry in Three Dimensions
Since this is my publishing space, I am going to spend some time on Mushinah Morris’ instructional and learning point here. Again, I was involved in the online teaching of chemistry for 14 years and my research speciality was science in VR, so I’ve got thoughts. If you would like to see her talk on video on this, she’s here in this video published by VictoryXR.
She is correct that molecules cannot currently be seen in everyday life. It makes chemistry, as a field, a more abstract or conceptual field along with physics when compared to the “you can see it before you” fields of biology or earth science. Teaching that something unseen exists and engages in reactions has always been the uphill battle of chemistry teachers. So she’s describing an accurate problem.
There is a tiny fly in the ointment, however, in that not many students at the college level fall into a chemistry course completely unfamiliar with chemistry at all. So learners in college chemistry probably were exposed to atoms in some other prior learning experience, be it high school, a museum, or a summer-camp like situation. So do her students need to learn atoms from the very beginning? I somewhat doubt that.
HCHE Advanced Inorganic Chemistry
To massively further complicate her argument, she names and shows her chemistry course “Advanced Inorganic Chemistry”. That’s HCHE 421 at Morehouse University, which in 2021 had a prerequisite:
HCHE 322 Elementary Physical Chemistry, which itself has 3 prerequisites:
CHE 321/321L, Elementary Physical Chemistry and Lab (which has 4 prereqs: CHE 232, PHY 154, and MTH 161 and 162)
PHY 253 Electricity & Magnetism, which has 2 prerequisites: PHY 154 (C or better) and MTH 162
MTH 271 Introduction to Linear Algebra, which has 1 prerequisite: MTH 161
You see where I’m going here. It’s highly doubtful that students arriving in an advanced chemistry class after what is years at college, whose content focus is actually math (that’s what inorganic focuses on) and not spatial abilities (which arguably organic chemistry DOES focus on) have a substantial problem with visualizing atoms and molecules to the point where it is disturbing their learning performance. And therefore VR could make a difference. No. Not buying it.
Covered in the mentioned course’s lab. That’s math, yo.
Nonetheless, I’ve known very smart and exposed people have trouble visualizing atoms. So it IS remotely possible and let’s pretend she is articulating only the beginning of the trouble of understanding for a lay crowd…not the only problem. Said another way, she’s speaking about VR’s affordances overall, maybe not specifically for her students in her aforementioned class. For example, some chemical reactions are easy to understand (like cooking) and some are difficult to understand (like how hair coloring works or cell electrical potentials).
It is interesting that she said “The learning tends to happen faster.” It’s a couched statement, for sure, with the word “tends”. In science that cannot be pinned down. So she gave herself an out. But what was she describing? At this point, we have to think about the instruction of chemistry.
How To Show Atoms and Molecules
Within the history of chemistry itself is the continuing saga of how will atoms be depicted? As in, how do you draw them? How are they really? And how does a teacher relay that ‘realness’ to the learners– and why?
So we’ve had our:
Atoms are indivisible tiny units, folks. Thank you to the Greeks! There are no pictures from that time.
We’ve had Bohr’s heliocentric-like model folks wherein the atom looks like a solar system or set of concentric rings. To be fair, the heliocentric model really does help explain things like electron energy levels.
We’ve had our Thomson plum pudding folks– which never translated from its culture. Which is probably a shame. I like plums.
We’ve had our ‘cloud model’ folks – which are like the postmodern philosophers of chemistry. Truth for me, truth for you, we all get a truth, which isn’t true. But they told us that electrons cannot be pinned down and measured, they could be anywhere at any time but when we set about measuring them, that’s when they run away from us. Yes, I’m nodding to Heisenberg here. And wave/particle theory.
After the heliocentric model, however, depictions of atoms needed to be displayed as three-dimensional, not just as two-dimensional on flat paper. By far, I’ve only selected some of the atomic model theories here. If you want to know more, study chemistry! It’s not hard.
But, now, going against Mushinah Morris’ arguments now, educators HAVE been working on that educational problem for years (with success, mind you).
First of all, delightful molecular (and atom) kits exist with physical manipulatives. Yeah, they look like tinker toys. I love them. They are good for at least 30 minutes of instruction, maybe more. They are usually plastic (boo, although there is nothing stopping them from being made of wood) and the kits would have to be purchased, stored, and de-germed from time to time. So they have their minor downsides.
Second of all, 2D screens can show 3D objects…that’s entirely possible.
Third, programming VR to follow mathematical principles – like, voila, chemistry DOES!– is actually not that hard. The first uses of VR in education that I know of were in the “physical”—that is mathematical sciences, physics and chemistry. Let’s face it. A computer understands 9.8 meters per second per second MUCH easier than a person does. (<- that’s one gravitational force).
And get this, purchasing a simulation to teach atoms is so drop-dead cheap that it’s actually free by now. I have recommended those simulations for courses before and seen learning scores do quite well, thank you.
Looks pretty 3D to me
I seriously bet that if I had been able to place that counter proposal before her administrators, I’d win the budget proposal. Ha! Bonus points that I could prove that my students would score equally to her VR students on the final exam.
So in all, did she make a good point here? I’d say no but that’s because I recognize the instructional problem and I realize that the problem can be solved in a much cheaper and equally as efficient way. Also, she showed no data that “the learners learned faster”.
Side point to the side point: Mushinah Morris on YouTube is highly associated with VictoryXR, the vendor that she is using when referring to her VR-for-education accomplishments. Close association with XR vendors makes for suspicious conclusions. I’m not picking on Mushinah Morris unfairly. She’s gone on the record multiple times for her claims. I could easily pick (and will in the future) other education influencers that are selling the VR-for-education snake oil.
Back to my article
Further, there is at least one study (so far!) that refutes this focusing-causes-faster-learning claim. Makransky, Terkildsen, et al. (2019) found that immersive metaverse environments could be sensory overload for learners and therefore decrease the learner’s focus. On the whole, claims for increased speed can often be attributed to more efficient instructional methods. Immersive experiences can allow for the utilization of comparatively faster instructional methods.
The author finds this myth, that immersive experiences cause learners to learn faster, false. (more…)
This conceptual series proposes instructional design principles for the metaverse. You’ve arrived at Part 3.
TL;DR
XR causes more learning and faster – Myth
XR is active learning – Myth
More immersion is better than less – Myth
XR causes greater retention – Jury is still out.
XR increases empathy – Danger Will Robinson Danger
Learners and instructors like it – True! But this means nothing to learning.
XR can impact far transfer – Jury is still out.
Positives get published – True, so negative or null results often do not.
Ready? Let’s do some myth busting.
Myths versus Reality
The metaverse in education is an emerging topic and a potentially lucrative field, possibly set to supplant learning management systems as the next industry-wide educational platform (Spilka, 2023). Improvements in technology are fostering an “anytime at anywhere” implementation of the metaverse (Tlili, Huang, Shehata, et al., 2022, p. 4). As interest in the metaverse as educational media has increased, misleading claims or myths have already circulated. These myths are often shrouded under the title of research until the curious probe a little deeper. This part will examine claims such as the metaverse will cause learners to learn more and faster, it represents active learning and is therefore better, it is more immersive than ever, learners retain more, it increases empathy, and learners like it so therefore they learn better. This section will mention areas where the research is still unknown, publishing bias, and what to look for when IDs read educational research on the metaverse.
1 More and Faster
Faster. You keep using that word. I don’t think it means what you think it means. (Note: this is also a woeful use of a time chart; three different times do not add up to a pie.)
Learners in the metaverse will learn more and faster; this is the first claim examined. IDs should maintain a healthy skepticism of claims that a certain media causes dramatic learning improvements. Claims often do not communicate instructional methods as Beck, Morgado, and O’Shea (2023) pointed out. Instead, current publishing focuses on outcomes-based research or what Reigeluth and Honebein called research-to-prove results that are “typically operationalized by comparing a new medium to a traditional one” (2023, p. 2). For instance, a lesson in XR could be compared to a lesson in a textbook. Similarly, the National Academies of Science, Engineering, and Medicine advised that possible external validity of some studies is low in that “there is considerable evidence that a single instructional technology can lead to different outcomes when used by different learners in different contexts” (2018, p. 194). These claims usually represent the pitting of two very different instructional methods and thus cognitive workloads against each other.
Here is an example of this claim. In a study incorporating virtual reality (VR) headsets for soft skills training, Scott Likens of PricewaterhouseCoopers claimed, “We found the realism and performance feedback in virtual reality simulations helped people learn faster and retain more information around soft skills,” (Zielinski, 2021, para. 9). However, the accompanying published report contradicted these claims. When comparing information retention in VR versus an e-learning course, the authors “quickly discovered retention scores were inconclusive, as the delta between pre and post-assessments in each modality was not significant” (Eckert & Mower, 2020, p. 44, emphasis added). Thus, the two different media showed no different learning outcomes.
Another claim is that the isolation effect of a headset causes faster learning, perhaps arguing that less distraction equals more focus. In the same study, Likens stated, “A lot of courses that normally take an hour could be completed in 20 minutes through VR because people are so immersed in scenarios, there are fewer distractions and the learning is very concentrated” (Zielinski, 2021, para. 10). Referring to the same study, “VR was x4 faster than classroom and x1.5 faster than e-learning” (D. Clark, 2022, p. 190). Claims that learning is completed faster attempt to represent XR as a more efficient learning method, i.e., less time to learn equals learning faster. When compared to classroom learning, it is already known that 1:1 personalized learning is faster. In this case, the classroom learning was allotted to two hours and the VR experience took 29 minutes. Given that 29 minutes is approximately one-quarter of two hours, the touted line was that XR was four times (4x) faster. In fact, the XR media did not cause the learning to be completed faster, it was the 1:1 nature of the learning experience.
[Editor Heather here: this is the same study I wrote about extensivelyhere and here and my colleagues wrote about here, in case you want to read more.]
Further, there is at least one study that refutes this focusing-causes-faster-learning claim. Makransky, Terkildsen, & Mayer have found that immersive metaverse environments can be sensory overload for learners and therefore decrease the learner’s focus (2019). On the whole, claims for increased speed can often be attributed to more efficient learning methods.
Lauding the media that manipulated instructional methods hides the fact that the learner could achieve the same results in a different media, given comparable time and resources.
This is a 4x claim variant, with “recall up to 400%.” They throw in the ‘control the distractions’ claim too. Upon request, the OP refused to provide a source.
[Editor Heather popping back in here about 5 months after this blog was originally published on 10/4/2023. I like to add examples “from the wild” when I see these claims and whaddaya know– a claim popped up yesterday! Here you go! Feast your eyes on the 4x claim along with the lack of reference/evidence/substantiation:
This particular example of the 4x is buried with a pro-Microsoft Teams article that I actually agree with (and posted another blog about here: https://heatheredodds.blogspot.com/2023/12/youll-be-using-xr-in-2024-and-you-will.html )
I’m curious that the 4x was applied to “retention rate” and “attention span” and was compared to Teams or Zoom, which, to the best of my knowledge WAS NOT in the 4x PwC study.
I feel some, ahem, elaboration has occurred here. And I find it interesting that while propping UP Teams (because it is rolling out immersive team meeting environments) this paragraph highlighted actually disses Teams. I’m thinking this person is so excited and into rolling out stats that he’s confused stuff.
2 Active Learning
Claim: Being in a headset equals active learning.
Some claims state that learner-instigated avatar movement, in the form of moving hands, heads, or bodies, or the first-person point of view makes XR learning inherently active as opposed to passive. Intentional avatar movement is associated with manipulating content, which is the term embodiment or embodied learning (Johnson-Glenberg, 2018; Markowitz, Laha, Perone, et al., 2018). The claim begins with the given that active learning is known to be better than passive learning. Because XR is body-movement active, it must be active learning and thus cause more learning (Johnson-Glenberg, 2018). However, research has shown that while embodiment does have a connection to learning, it does not exclusively cause learning. Truly, “platform is not destiny” as Johnson‐Glenberg, Bartolomea, & Kalina stated in 2021 (p. 20). Just putting a learning experience with movement into XR does not make it active learning.
3 More Immersion
There are some claims that take issue with the second assumption stated earlier in this series: Learning outcomes are expected to be equal to other media (Mayer, 2020). These claims state that earlier comparative media studies did not show improved results because the technology then was older. Thus, technology now utilizes a better quality of immersion. This claim reflects a modernist philosophical approach: newer is better. Cummings and Bailenson (2016) reported that head tracking, stereoscopic visuals, and wider fields of view created more immersion than other visual or audio improvements. Yet Mayer wrote in 2020, “these comparisons between low-immersion and high-immersion media do not provide strong evidence for the instructional value of converting a 2D lesson rendered on a computer screen into a 3D lesson displayed with a head-mounted display in immersive virtual reality” (p. 365). Moreover, Abbas, Seoo, Ahn et al. (2023) found that high levels of presence did not impact user behaviors. Ochs and Sonderegger (2022) reported that learners that felt an increased sense of presence in VR scored worse on measures of memorization even when the learner simultaneously self-reported that they expected to do better in VR versus 2D. Finally, Makransky, Terkildsen, & Mayer found that VR causes more presence but less learning (2019).
Overall, these claims also fail to acknowledge that the main subject in media studies are humans (not the media), and we already know a great deal about how humans learn in 3D environments. These spaces exist outside of technology and are called classrooms. These claims that newer XR will cause more learning look more like calls to buy the latest technology.
That is not instructional design research, it is marketing.
4 Greater Retention
Claim: better retention, “it has been proven”. Not so much.
This claim states that XR enhances knowledge retention (Victor, 2023). Studies of retention are still ongoing and difficult to find. Indeed, broad reviews such as those conducted by Hamilton, McKechnie, Edgerton, and Wilson (2021) commented that finding “learning outcomes, intervention characteristics, and assessment measures associated with immersive virtual reality use has been sparse” (2021, p. 1). This aligns with Beck, Morgado, and O’Shea who contended that, “Very few literature reviews focus on the educational practices and strategies used in immersive learning environments. Thus, the problem is that we are evaluating outcomes without a comparable way to describe the educational approaches that led to those outcomes” (2023, p. 2). Therefore, retention could be achieved with XR implementation, but without more research detail, greater results might be attributable to the method, not the media.
The use of the metaverse in education is not yet common. It is difficult to find studies that measure retention within learners more than 10 to 21 days after instruction. Practical workplace implementation would require much longer retention times. Therefore, this claim has not yet been supported or refuted.
5 Increase Empathy
Someone seems to think so.
Research on empathy indicates that this is an area of risk. Because of a first-person point of view in many XR experiences, learners perceive a direct impact of the experience which is meant to foster empathy. However, empathy, like presence, is nuanced. Indeed, in some empathy research, learners did not react with a positive and caring response, but instead with disgust and rejection (Bailenson, 2018). Thus, the objective of the experience might be not only missed but soured.
Clark, when writing about accessible pedagogy in immersive learning, advised to avoid first-person depictions of marginalized groups because XR experiences cannot portray the depth and spectrum of a person’s life. “Instead of teaching students what it’s like to be blind, consider having them deconstruct the ways vision is assumed in how spaces are designed, as well as the ways their understandings of vision impact how they interact with others,” then “focus on bringing awareness to the assumptions built into the physical world around them” (J. Clark, 2021, Recommended Administrative Considerations section, para. 4). Therefore, XR to foster empathy should be approached with extreme caution.
6 Learners and Instructors Like It
Forward-looking statements of optimistic activity. Unfortunately NOT strongly connected to learning.
This learning claim, that learners liking a learning experience will then learn more, is perhaps the most common. To the contrary, there is no research-based connection between liking an experience and learning success (Hughes, Gregory, Joseph, et al., 2016; Uttl, White, & Gonzalez, 2017) . Thalheimer further discounted the connection between learners liking their learning and achieving their learning, when he stated that “measuring interest is an inadequate way of measuring learning” (2018, p. 26). Therefore, while it is pleasant for learners to enjoy an experience, it has no firm connection to a learning outcome. Beyond the learner, education professionals should use caution when thinking that emotional motivation will work over the long term. Instructors often become enthralled with possibilities of XR and start to believe that the feeling of being there (a combination of presence, embodiment, and immersion) will make a positive difference. Research and theory are not forecasting this. However, the flame of excitement among professionals should not be extinguished. Clark and Mayer advocated for a tempered approach where the ID can keep to the best learning practices and not be distracted by the media:
“The challenge in e‐learning, as in any learning program, is to build lessons in ways that are compatible with human learning processes. To be effective, instructional strategies must support these processes. That is, they must foster the psychological events necessary for learning. While the computer technology for delivery of e‐learning is upgraded regularly, the human side of the equation—the neurological infrastructure underlying the learning process—is very old and designed for change only over evolutionary time spans. In fact, technology can easily deliver more sensory data than the human nervous system can process. To the extent that attention‐grabbing audio and visual elements in a lesson interfere with human cognition, learning will be depressed” (2016, p. 24, emphasis added).
7 Near Versus Far Transfer
Claims about near transfer outcomes will be addressed just ahead [look for Whitney]. Results showing positive far transfer from XR applications, however, are elusive in a similar way to the retention results. Research shows equivalent or mixed performance to traditional media (Kaplan, Cruit, Endsley, et al., 2021; Makransky, Borre‐Gude, & Mayer, 2019) or worse performance (Makransky, Terkildsen, & Mayer, 2019; Parong & Mayer, 2018). In particular, Mayer offered the Immersion Principle in 2020 which stated: “People do not necessarily learn better in 3D immersive virtual reality than with a corresponding 2D desktop presentation” (p. 357).
Intuitively, because XR can replicate real world environments where the learning would be applied, far transfer seems like a reachable goal. Tlili, Huang, Shehata, et al., 2022 wrote that the technology can enhance and allow for transfer. Johnson-Glenberg noted that despite requests for more research into XR, “resources and affordable technologies were not readily available” (2018, p. 7) for educational research and that “longitudinal effects of VR exposure are unknown at this point” (2018, p. 11). It is possible that not enough time has passed for the research community to measure the ‘far’ in far transfer. In general, the ability to do worked practice exercises repeatedly in simulated real-world contexts suggests that XR should be at least equivalent when compared to other media for far transfer.
8 Positives Get Published
Research with positive results is published more often than research with negative or no results. This is not unique to metaverse applications. This is known as publication bias or the file drawer problem (Lederman & Lederman, 2016). It limits the results of a meta-analysis because if a particular form of learning is not effective, it usually is not published (Cofré, Núñez., Santibáñez et al., 2019). Thus, the published collection showing positive results with XR dominates over the no significant difference results.
As an added caveat, IDs should closely examine research funding sources and sponsors.
How Will I Know?
You are welcome for that earworm. Who doesn’t love Whitney, helping out with instructional design?
IDs are cautioned to examine metaverse research studies for these two major characteristics:
Novelty effect. The novelty effect is when learners are exposed to a new media and they engage in increased effort and attention. It tends to positively impact learning outcomes (Metcalf, Chen, Kamarainen, et al., 2019) but not always (Huang, Roscoe, Johnson‐Glenberg, et al., 2021). Further, “studies of virtual reality-based learning are based on only short-term implementations, and although they might show statistically significant learning outcomes, the novelty effect is an important caveat to the research because many of these studies do not account for the decay of outcomes over longer periods of time.” (Metcalf, Chen, Kamarainen et al., 2019, p. 97)
If a research study implements a one-time 20-minute XR intervention and claims to show learning improvement, the learners are likely experiencing the novelty effect.
Non-cognitively comparable methods. Studies where the learner is not put into the same cognitive workload with two different media should be viewed with skepticism over claims of better results (Reigeluth & Honebein, 2023). For example, if a study stated that learners performed better in XR than paper-and-pencil-based learning, the results should be discounted due to the varying cognitive impact that the different media had on the learner (Parong & Mayer, 2021). In one example, the experimental learner group was exposed to VR training after the standard training and then scored higher than the control group (Seymour, Gallagher, Roman, et al., 2002). The total training time increased. This could have caused higher scores. The two conditions, therefore, were not comparable. Furthermore, the National Academies of Sciences, Engineering, and Medicine noted that the prevalence of WEIRD populations (Western, educated, industrialized, rich, and democratic) used in educational research inherently exclude diverse learner populations and this makes it difficult to draw solid conclusions for all humans in all learning situations (2018). Thus what works for one group of learners might not work, nor even be comparable, for another group.
Clark and Mayer summarized how to examine research claims for e-learning, but these questions equally apply to XR research.
“Are the methods, content, learners, and context like yours?
Does the experimental group outscore the control at a significance level of p < .05? [Editor Heather here: How many of y’all KNOW what the phrase “statistically significant” means? I thought so. I’ll write a future article on it so that you stop banging the “XR will make a significant difference in education” phrase around. I hate that. Be warned.]
Does the effect size favor the experimental group at a 0.5 level or higher (2016, p. 63)?”
IDs will likely encounter innovators and early adopters who have anecdotal stories of how XR improved learning.
A passionate-for-VR educator describing what is likely the novelty effect with their students.
These stories should be accepted with grace, as every form of media has the possibility to hit the perfect instructional moment with the right learning at the right time for the right learners.
I sprinkle just the right instructional media and methods here and PERFECTION.
Longer term and wider implementation decisions, however, should be made more systematically, by thinking and rethinking the design decisions over time. IDs rarely have control over the large financial decisions that XR development requires, so their role can be one of consultant: offering all the options and pros and cons of each to the decision makers (Dodds, 2021). After reviewing research, IDs are ethically bound to point out if a learning objective can be met with a cheaper, more environmentally responsible, or more socially just media.
In summary, “As a consumer of experimental research, you need to be picky!’ (Clark & Mayer, 2016, p. 56)
Part 4 will answer “How do I know I’m on the right trail with this [assigned] XR project?” (Yeah, more Whitney!)
My first article in this series garnered so much attention! But many folks tried to pass me Mirjam Neelen & Paul A. Kirschner’s Truth or Truthiness? Analysing a VR Study Using Gorard’s Sieve article on the PwC report entitled “The effectiveness of virtual reality soft skills training in the enterprise: a study” and all of its associated webpages like this one. I was like, I know! Mirjam & Paul wrote their article 2020 and I wrote about it in 2021. What’s cool is that separately, we both came to the same conclusions. That’s a good sign for our conclusions!
Short version: we both cast strong doubt on any conclusions.
Still, I realize the world does not revolve around me (sigh!). Some folks might have missed my long stream-of-consciousnesses article about the PwC report. I decided that the second article in this series should be an abbreviated and updated critique. Bear in mind that to reach the LinkedIn audience, I have to leave much nuance by the side of the road. If you have questions, just ask!
As Mario says “Here we go!”
What is Said About The Report
This infographic summarizes the dominant conclusions:
275% more confident to act on what they learned after training
4x faster than classroom training on average
4x more focused than e-learners
3.75x more emotionally connected to the content than classroom learners.
Capture of how the PwC report is being talked about on LinkedIn.
A few more quotes, thanks to Google and a search on “VR 4x faster.” What seems to be a pattern about all of these results?
What do these Google results have in common?
They are all companies that sellsome sort of VR product or service.
Because I was curious, I checked out that vrowl dot io link (“Virtual Reality training is not effective”) just to see if it was presenting an alternate opinion. It’s a strawman argument; it puts up “not really real” protests against VR for learning and then explains them away. I’m telling ya, Beware the VR Strawman.
Let’s ask Google Scholar what it thinks. It’s coming up with 11 cites. That’s not much at all. But as I showed above, the money shot is on the Internet, not in academic articles.
Truly, the 4x faster learning quote is the runaway train of this report.
There you are, eating your morning bowl of cereal. You glance at your social media feed. Up pops these images:
Wow. Even the infographic people seem amazed.
But
if it is too good to be true, it probably is, at least where research
and marketing intersect. Not to worry! I’m here to tear apart this and
see what’s inside. Can virtual reality (VR) teach soft skills training?
Verdict: Maybe.
My Executive Summary:
Let’s start with the nuts and bolts.
Citation & Report
Mower,
Andrea. “The effectiveness of virtual reality soft skills training in
the enterprise: a study”. [Place of publication not identified]:
PricewaterhouseCoopers, 2020. Online. Internet. 21 Aug 2021. .
Available: https://www.pwc.com/us/vlearning.
Despite that
academic-y looking citation, the writing does profess itself to be a
report, not research. That’s good because reports are not held to the
same standards of rigor as research.
What you find at that web
link, however, is NOT the full report. You are looking at the corporate
summary. Remember that technically, PwC is not in the VR business (a
plus) so they are not selling you something about VR. They are only telling about how some VR training went at their company.
Disclosure is right up front (first sentence!) that:
“supported by Oculus for Business and Talespin,”
Good
disclosure; it’s good practice. While I’m not loving that it is
sponsored by a VR headset manufacturer and training creator, knowing
this lets me view this with the appropriate amount of critical thinking.
Their study started in 2019 and ended in February 2020, so they indicated when it was run.
Note:
Every piece of research that touches 2020 and forward into the near
pandemic future, should clearly mention WHEN the study was run, because
the COVID pandemic is impacting every part of our lives. COVID does impact “reports” and we need to know if these are ‘at-home-stressed-but-sent-a-headset’ users.
They had a hypothesis: Our
hypothesis was that training using VR is more effective in achieving
learning outcomes than traditional training methods (classroom or non-VR
digital experiences).
Remember that a
hypothesis in experiments is good. Hypotheses guide us to our data and
results. Bias in experiments is bad. Bias makes us ignore our data and
results.
And PwC defined “more effective” with:
Employee satisfaction
Learner flexibility
Comfortable learning environment
Improved attention
Higher information retention
Confidence building
That definition of “more effective” is a little murky. Usually time plays a very definitive role in “effective” measurements. For example: widgets produced over time. Here, time is not actually mentioned and yet time is prominent in the infographic stats later. Hmm..
In their experimental design, PwC appears to think that they have made comparable training:
classroom,
e-learn,
v-learn (VR).
Oo, bust here. While I hat tip to the innovative thought process expressed here, to take advantage of what VR can offer…
“The
classroom and e-learn course experiences were linear: A video was
shown, the learners asked some questions, then the next scenario was
presented.”
“However, we determined this linear approach would not
leverage any advantages of the VR modality. We hypothesized that
placing the learner directly in the scenarios covered in the curriculum
and giving them the ability to act as they might in real life would be
more rewarding for them.” (p. 16)
… but, owch, non-comparable methods!! If you literally taught the information differently, you cannot compare the methods and thus, you cannot compare the results.
The key to making a good design that includes VR (and I’m giving you
the $64,000 answer here for anyone designing “compare” research with VR)
is to put it up against something very, very cognitively similar.
Right now, that technology is 360 video where you put in branching
decisions. And add haptic bodysuits & controllers. But that convo
is for another day.
Now to parse out the difference between 360
video and VR, you’ll need thousands of users. Not many investors right
now have the willingness to spend thousands of dollars in equipment and
time to collect that much data. And even when you get there, you
probably won’t find much a of difference. Why? Well, take a look at what
you designed. If you make a cognitively similar experience and run
humans through it, it actually makes sense that no significant difference in the data will arise.
Remember at this point, I’m talking about learning outcomes ONLY, no
other characteristic. Also, go back and look at the hypothesis. They
already thought that the ‘learning outcomes’ would be the same (READ:
same scores on tests) but that they could achieve those learning
outcomes “more effectively”. That’s interesting. I wonder how much of
this report was written from hindsight and how much was written before
the study started. (This is why you write your hypothesis first–before
your study.) That hypothesis is now possibly showing a bias…did they
*think* it was going to turn out…the way it did?
Number of participants: 1600 possible. Good. But they never disclose how many learners they actually had in each group (their Ns).
Therefore, I’m going to call this a strong negative because they could
have included that number (I don’t see a business reason to conceal
it).
Experiences: 5-7 minutes long
Did use Oculus Quest (ahem)
Did use Oculus for Business for remote device management
Honorable
mention in the report: Using the phrase “not the most gratifying” when
talking about tagging and inventorying what must have been more than 300
pieces of equipment.
One more small note: The report does a nice
job explaining how they decided to buy 100 headsets and what the cost
and time of developing the v-learning training was. It is outside of my
scope to analyze that but I will recommend it as good to read. Remember
that if the cost of v-learn is dropping, it becomes a better and better
choice over time. They do those calculations and find that it becomes
the better choice at 3,000 users/learners.
Let’s see what the data shows:
4x faster to train than in the classroom
I
might have to cede this one right off the bat because VR does provide a
1:1 experience that most classrooms cannot beat. How many classrooms
can provide– minute for minute– the same 1:1 attention of the teacher
to the student? Erps. Few.
Fly in ointment? VR can cost a great deal of money for development and for the equipment. READ: the set up.
Counter to that?
VR can be done for very little money and because training can be
replicated a billion times when a 1:1 teacher cannot be replicated, VR
wins the day on this claim. READ: it CAN save time but after
development is done.
I still don’t like comparing “classroom” to VR, such a not-fair comparison.
“What
took two hours to learn in the classroom could possibly be learned in
only 30 minutes using VR. When you account for extra time needed for
first-time learners to review, be fitted for and be taught to use the VR
headset, V-learners still complete training three times faster than
classroom learners. And that figure only accounts for the time actually
spent in the classroom, not the additional time required to travel to
the classroom itself.”
In the report, they shared numbers in minutes.
Classroom:
2 hours (Watch that crossfire, boys! They didn’t express this as 120
minutes. Heather gives PwC a strong look for that. Don’t be mean. Or
get a better editor.)
E-learn: 45 minutes
V-learn: 29 minutes
29 * 4 = 116
116 is close to 120. Therefore, saying V-learn is 4 times faster is accurate. [Edit: In future writing, I clarify that stating “4 times” or “4x” is not a data lie, but it expressing a truth in a deceptive way. Learners didn’t learn four times faster, the literal training session was 1/4 as long in time. So they sat through LESS training time. That doesn’t mean that they learned faster.}
275% more confident to apply skills learned after training
This is a poor item to measure when we are focusing on learning outcomes. The Dunning-Kruger effect says that those least able to accurately self-measure something are actually the worst at that thing.
They
make an argument that particularly with soft skills, confidence AFTER
training would help implement the soft skills in the workplace. I liked
their design inside the V-learn module. Learners had to say their lines in the simulation. Nice touch!
But alas, this is all future prognostication and not actual data. We can hope for something but that doesn’t mean that our hope leads to actual results.
166% and 275% of what? Where is the 100% in this diagram? Said another way, what is the baseline? Zero?
I still can’t find this in the report.
3.75 x more emotionally connected to the content than classroom learners
I’m
not going to pick this one apart much because the errors should be
apparent. Often VR presents training as first person– meaning the user
looking through the headset is often the protagonist of the adventure.
Therefore, a story happening personally to the user creates more emotional connection than the same story in a classroom. This comes from the bucket labelled obvious
and is frivolous data. It also blatantly shows what’s wrong with using
non-comparable instructional designs. This is an apples to oranges
comparison.
FURTHER, newer data is showing that types of empathy count…not just general empathy or emotional connection.
4x more focused than their e-learning peers
There
are some great accessibility studies and autism spectrum studies coming
out that are showing us some very interesting research (really, it’s a watch this space
stuff) that VR can be more cognitively overwhelming for some learners
and hence every measurement that says learners have more brain
engagement could actually be learners overwhelmed (yeah, obvious bucket
again).
But let’s look at what was actually said here:
“With
VR learning, users are significantly less distracted. In a VR headset,
simulations and immersive experiences command the individual’s vision
and attention. There are no interruptions and no options to multitask.
In our study, VR-trained employees were up to four times more focused
during training than their e-learning peers and 1.5 times more focused
than their classroom colleagues. When learners are immersed in a VR
experience, they tend to get more out of the training and have better
outcomes.”
It doesn’t say how the “more focused” was measured? I
wonder? Remember the clue to look closer is when the presentation tends
to do a ‘hand wave’ approach on something. It’s equivalent to
misdirection by a magician. If you look over here, you are not looking
over there. The text says “There are no interruptions and no options to
multitask”. Yes, the Oculus Quest headset design doesn’t allow a user
to look around or to (reasonably) be interrupted by messages from outside of
the experience. But did that mean interruptions didn’t happen? What
about a “this is pulling my hair” message? Or “it’s fuzzy” or even “I’m
gonna puke.” Those are interruptions caused from the inside out.
Were those counted?
In the report, it looks like self-disclosure:
At
this point in the paper (p. 45), my hackles are going up because you
cannot see data (self-reported), acknowledge it (we thought it was
higher), then throw it out (“the statistic was significant and should
result”??). What? PwC? My doubt meter is red-lining at this stage.
Insert Mr. Potato angry eyes.
5. VR learning can be more cost-effective at scale
Interestingly,
this didn’t make an infographic!! What?? What a shame because right
here I AGREE with this paper!!!!!!!!!!!! ARGH!~!
YES, YES, YES. VR
is more expensive to make once (but that cost is dropping) and it can
be replicated (which is where you win) but it is also showing
impressive results in FLEXIBILITY (meaning, you can change up the
conditions quickly). VR can be made cheaply, which also means basically,
but that might not be a bad thing. Get over the hump of the novelty
effect and design a basic experience which is accessible to many
learners and you are in an effective horse race with other forms of
learning. What I’m trying to say is that basic and/or cheap isn’t
necessarily bad in VR. It should not be thrown out. Because this is
where VR is going to eventually win.
They are using their own
costs and admittedly, the fact that it’s a large study now hurts them
because that means more money was outlaid to get the training started.
Revisiting how many actual participants???
They said 1600 eligible but they never said how many it was AND then
they said that they “offered” v-learn to the classroom and e-learn
participants as an option (and those went on to answer a smilie sheet on
how much they liked it).
One Result Buried
OMG look at this that they tucked into the back of the paper, I didn’t even know it was there!!!!! Page 44.
PwC study conclusion: No significant difference in retention
No SIGNIFICANT DIFFERENCE in RETENTION!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I’m
currently on the floor laughing because that is what research data has
lead us to predict all along. No difference. They’re blaming the
content in this case. OK, picking myself up off the floor and dusting
myself off. Insert droll look here. It’s not the content, yo.
What PwC is missing right here is this: No significant difference results eventually means that the cheaper option to learn the same material will be market dominant. Translation: make VR easier to MAKE and VR will be bought. Cha-ching.
Final verdict: Mixed Bag
•2 metrics (confidence, focus) = garbage
•1 metric (faster) = just okay
•1 metric (cost effective) = good
•1 result (no significant difference) = buried
Conclusions
• Enough doubt to worry about other PwC infographics
• Overall, not bad for a “report”
• Not research.
Can
VR teach soft skills? It appears so, yes, at least as well as
classroom and elearn options. (No comment on the quality of the
instruction or the assessment.) Right now, v-learn is an expensive
choice. But the price for development IS dropping. I have high hopes.
This was my 3rd planned article on analyzing research.
More
to come in the next few days because the next article is already
written. That will end, hopefully (!), my series on poor VR &
learning research. [Edit from the future: The bad research as kept on coming. My “Seeking Integrity” Series refreshes this topic.]
This is a copy of the same article that I posted to LinkedIn on August 25, 2021. This post was slightly edited on April 11, 2026 with an improved font, re-placed banner image, and added link to further writing on this PwC study.
.png)
