Given that it looks like EDUMetaverse is headed to the turf, I might as well get a blog post up prognosticating that ENGAGE XR is going down too. I might as well. I’m not even early to these thoughts; I heard through the grapevine that ENGAGE XR was laying off folks within the past 12 months. I did research that and it is true.
LinkedIn shows negative 41% employee growth in the past 12 months.
Not aging well? Morehouse College, ENGAGE XR, and the Metaversity
So this is a case of the chickens coming home to roost. One can advertise all one wants to about how great one’s XR is, but if one is propping up untruths, failure will follow.
From Myths to Principles: Navigating Instructional Design in Immersive Environments Part 7 Myth: Immersion creates empathy
I’ll be the first to point out that my blog posts are not published regularly. As I’ve mentioned before, this series is an updated version of an older series. But these first four myths are mostly ground I’ve covered before. Additionally, since the myths are just basic lies, it’s really hard to muster the motivation to write about them AGAIN.
Every time I have a difficult project, I weigh up working on it versus cleaning the toilet: which would I rather do? Yo, the toilet is pretty clean. So…these blog posts have not been winning that decision.
I’m truly in the dark part of the woods on this entire topic.
But what makes me continue? I’ve said it before and I’ll say it again:
The lies keep being repeated.
In the past week [EDIT: I wrote this on February 27, 2025], I’ve heard:
In a sales pitch to a school to use VR, that there are (proven?) tangible learning outcomes.
When learners were using VR headsets, engagement happened.
I’m reminded of a baseball quote that I used to motivate my team when they were having really long hard days at work:
âYou win a few, you lose a few.
Some get rained out.
But you got to dress for all of them.â
â
Satchel Paige
You’ve got to suit up for them all.
I take from this, that choosing to enter the arena is more important than the outcome of the arena. I may loose the war against false, malicious, money-making claims about XR in education, but the important point is that I chose to speak out.
So here I go, suiting up for our final myth of this series: Immersive experiences are empathy-generating experiences. Here we go.
From Myths to Principles: Navigating Instructional Design in Immersive Environments Part 5 Myth: Learners learn more
The
foundation of all learning, or child’s first book, by which a child
will learn more in one month than by many others in twelve, Author Unknown, Date 1800. Source: Compositor, University of Birmingham
Myth: Learners learn more in immersive experiences
This myth shrouds itself within a cloak of research. Citations will state that learning in immersive experiences is somehow greater when pitted against an implied traditional learning approach. The claim could appear as retention, but it is related to how well the learning was accomplished when measured up against learning objectives or a final goal.
When referring to the PricewaterhouseCoopers (PwC) report results mentioned earlier in this article series, Scott Likens claimed, âWe found the realism and performance feedback in virtual reality simulations helped people learn faster and retain more information around soft skills,â (Zielinski, 2021, para. 9). He claimed they retained more information. This specific claim has been repeated in academic literature, which stated, âStudies have found that students who use XR training are more engaged with the content, display more confidence with the material, and retain more information than students who use traditional training methods.â (Rode, 2024, para 2.)
A close examination of the PwC report, however, reveals that the claim was unsupported within the reportâs own data. When comparing information retention in VR versus an e-learning course, the authors âquickly discovered retention scores were inconclusive, as the delta between pre-and post-assessments in each modality was not significantâ (Eckert & Mower, 2020, p. 44). Thus, there was no statistical difference between VR-based, e-learning, and traditional classroom learning.
Claiming
something happened but your instrument didn’t pick it up is the knomes-did-it territory of cause-and-effect, dudes. Watch out.
The report therefore does not provide statistical evidence of more or greater learning within VR, yet it has been cited in academic publications (O’Dwyer, 2021, Etienne et al., 2022, Jelki et al., 2022; Bäckelin, 2023; Etienne et al., 2023; Lønne et al,. 2023) and touted in media outlets (Murad, 2023; Schwantes, 2020). For the dubious claims, the report has been debunked as untrustworthy (Neelen & Kirschner, 2020).
There are similar claims about greater learning retained from immersive experiences. Advocates for digital twin campus environments claimed that they âcreate greater retention of the information that is learnedâ (DâAgustino, 2022, para. 5) and âstudents’ grades go upâ (Victory XR, 2024, Who We Are).
While this is just one tiny sentence, keep in mind how much money VictoryXR makes from these claims.
In another example, the CEO of the Miami Childrenâs Health System touted that learners had 80% retention after one year after using VR, but traditional learners had 20% retention. A close look at the supporting documentation shows that the CEO actually said that the difference between VR learners and traditional learners can be the 80% to 20% difference.
However, the CEOâs statement did not refer to any published results; it was opinion. The CEO explained their claim by saying that, âThe level of understanding through VR is great because humans are primarily visual, and VR is a visual formatâ (Gaudiosi, 2021, para. 4). Cue learning styles!
Nonetheless, the quote of 80% retention has made its way into academic research (Iacono & Vercelli, 2019; Mathew & Pillai, 2020; Ternès, 2018). Some claims are extreme. One keynote speaker, Alvin Graylin, speaking as a leader at HTC (a VR headset maker) declared that as a result of VR use in the classroom, âEvery single child has the potential to be a geniusâ (Educators in VR, 2020, 23:33).
“Every single child has the potential to be a genius.” You just need to buy a (HTC) headset.
Justification for the greater learning or retention claims seems to be conjecture. Claims refer to how real an immersive experience feels to a learner. Returning to the PwC report, Likens credited âthe realism and performance feedbackâŚhelped people..retain more information.â (Zielinski, 2021, para. 9). But long term studies measuring retention are hard to find within the body of academic literature. Some studies measure retention three weeks after the immersive experiences. Given that many of the examples of immersive experiences relate to hands-on disciplines like nursing or construction, things learned in immersive experiences would be needed on the job more than three weeks after the training.
Hamilton et al. (2020) stated that finding âlearning outcomes, intervention characteristics, and assessment measures associated with immersive virtual reality has been sparseâ (p. 1). Beck, Morgado, and OâShea (2023) pointed out that details of methods are missing so that outcomes become questionable, âVery few literature reviews focus on the educational practices and strategies used in immersive learning environments. Thus, the problem is that we are evaluating outcomes without a comparable way to describe the educational approaches that led to those outcomesâ (p. 2). Lawson et al. (2024) completed a systematic review of immersive experiences and found that research studies rarely isolate instructional methods and conditions when describing research studies and thus impact real world classroom decisions.
Meta analyses are starting to illuminate this area. AkgĂźn and AtÄącÄą (2022) observed that there was only a moderate effect on learner achievement after surveying 31 studies. Kaplan, Cruit, Endsley, Beers, Sawyer and Hancock found that âXR does not express a different outcome than training in a non-simulated, control environment. It is equally effective at enhancing performanceâ (2020, p. 1) Some researchers are starting to incorporate machine learning and artificial intelligence into this challenge in order to determine what the published records states about immersive experiences. Markowitz et al. (2024) recently surveyed 196,734 paper abstracts with this method.
Basically
this graph says that because the numbers are so small and so close to
zero, there is no discernible effect of VR on learning, regardless of
immersion (Kaplan et al., 2020).
I’m starting a new article series today, calling out ‘bad research’ or research that is quoted badly in virtual reality for educational use. I thought I would start with a whopper – a really egregious example to start this series with a bang. Then I checked my notes and realized that this example is from LAST MONTH, June 2023. I’m not even going into the vault for this. I’m barely picking myself up off the ground from the shock wave.
So, like Mario says “Here we go!”
What Is Said About The Research Versus What The Research Says
June 2023, LinkedIn Post:
“According to a study from the University of Maryland in 2018, learners remember an astounding 90% of what they experience in VR compared to merely 10% of what they read and 20% of what they hear.”
I believe this is the research referred to:
Krokos, E., Plaisant, C., & Varshney, A. (2019). Virtual memory palaces: immersion aids recall. Virtual reality, 23, 1-15. https://obj.umiacs.umd.edu/virtual_reality_study/10.1007-s10055-018-0346-3.pdf
Hey, I’ll give you the abstract because I know you don’t like to read long papers:
“Virtual reality displays, such as head-mounted displays (HMD), affords us a superior spatial awareness by leveraging our vestibular and proprioceptive senses, as compared to traditional desktop displays. Since classical times, people have used memory palaces as a spatial mnemonic to help remember information by organizing it spatially and associating it with saliďż˝ent features in that environment. In this paper, we explore whether using virtual memory palaces in a head-mounted display with head-tracking (HMD condition) would allow a user to better recall information than when using a traditional desktop display with a mouse-based interaction (desktop condition). [OK skip to here because this is the interesting part:] We found that virtual memory palaces in HMD condition provide a superior memory recall ability compared to the desktop condition. We believe this is a frst step in using virtual environments for creating more memorable experiences that enhance productivity through better recall of large amounts of information organized using the idea of virtual memory palaces.”
Google Scholar tells me this study has been cited 461 times. That’s a low-medium citation number. Not bad, and remember that’s in ~3 years of time.
Believe it or not, I’m walking RIGHT PAST that 90%, 10%, and 20% because it has already be debunked here and here. Also, to be fair to the research paper, it never quotes those 10 and 20% numbers.
My Take on the Research
Research found 90.48% recall in the headset condition, with a 78.57% score from the desktop display control group. So thatâs ~10% higher with the headset.
From Section 4.1 “Using a paired t test with BonferroniâHolm correction, we calculated p = 0.0017 < 0.05 which shows that our result was statistically significant.”
Interesting. I’m not familiar with Bonferroni-Holm correction.Just looking at it, it appears to be a method of discarding some data. I wonder if NOT using it showed a not statistically significant difference between the 90 and 78. Their n was 40. Smaller group sizes means it can be harder to justify the data as fitting a normal bell curve.
Figure 5 shows the data and just looking at it, you can see that the numbers landed in similar scores. The boxes overlap, so whatever the effect of VR is, it’s not that substantial in this study. Students were learning, regardless.
But here comes the whopper. Check out this little detail in the Materials section:
“For this study, we used a traditional desktop with a 30 inch (76.2) cmâdiagonal monitor and an Oculus DK2 HMD. The rendering for the desktop was configured to match that of the Oculus with a resolution of 1920 Ă 1080 pixels (across the two eyes) with a rendering field of view (FOV) of 100âŚ. In order to give the desktop display the same field of view as the HMD, the participants were positioned with their heads 10 inches (25.4 cm) away from the monitor.”
10 inches away
The “control group” sat 10 inches from their desktop monitor to use the desktop condition.
WHO DOES THAT?
You know, I was curious. I grabbed my ruler.
How far away are you sitting from YOUR monitor?
I’m currently sitting 24 inches from my monitor. I leaned in to feel what 10 inches is like.
At that point, it became no wonder to me that the control group scored about 10 points lower. It was maddening. Remember, the learners had to look all around themselves so completing learning at 10″ from the monitor would be…uh…weird?
This is a great example of not seeing the forest for the trees in VR in education design. In order to match the field of view, they forced learners to unusually use their desktop monitors.
There is too much. Let me sum up.
The quote is from a keynote speaker at a research conference. I can’t believe anyone in the audience did not flag the play on the quote, the percentages, or the design setup of the U. of Maryland study. At the industry.
The difference between 90 and 78 *might* be too close to call a difference caused by VR.
Setting up learners to use a monitor from 10 inches away is unusual, to say the least.
When research sets up unfair comparison conditions, the results should be questioned.
As Hill Street Blues would say, “Let’s be careful out there.”
.png)
