In my continued series on how real companies are wrestling with using VR as an educational tool, today’s article focuses on managing costs and measuring success. These notions are based on a recent webinar entitled “VR & AR/MR for Training & Education – What have we learned?”, a digital offering from (XR) Intelligence by Reuters Events. My previous article in this series can be viewed here.
The speakers hailed from a variety of industries, including a children’s hospital, an industrial rentals firm, car sales/manufacturing, and B2B VR development, featuring:
- Jeffrey Jacobson (Boston Children’s Hospital)
- Walter Davis (Aggreko)
- David Moreno (Virtualware)
- Amanda Clarida (Volvo Trucks)
Managing the Production Costs of VR
Developing VR with a commercial partner or attempting the builds in-house can be an expensive proposition, according to the webinar’s speakers. Moreno suggested that the production costs can range from $50,000 to $100,000 (and even higher) for XR training instances. As a result, some insightful strategies for managing costs were briefly highlighted in this webinar. Some of the best takeaways for managing VR development costs downwards are paraphrased below:
“Always begin with both a business case, a starting budget and a sense of the production time desired in order to avoid the cost creep of VR development”.
“Determine the desired amount of collaboration with vendors along a sliding scale from going it alone to complete outsourcing”.
“Use offshore vendors; they are more reasonable”.
“Since C-level leaders have often never tried VR, start with some numbers, which they do understand: offer up a pilot trial aimed at 5-10% of the potential ROI expected for the project”.
“[Since smaller companies cannot afford the time or money to build Unity simulations from scratch] use Unity or Captivate in-house for proof-of-concept trials to keep costs lower”.
“Work with a vendor to provide a “proof of concept” development at no cost. (Ask the vendor to “take a little bit of your content and show you what it would look like” if you went the whole distance with them”.
A final suggestion emerging from the production cost discussion may be best summarized in my own words: in education, we often refer to the concept of “Use and Reuse”. Basically, this notion suggests that, if you aim to cost effectively develop VR instances, avoid starting from scratch each time; look for scenes, characters, or content that can be rapidly amended, ported over or “reused” in subsequent instances.
Measuring Success
Each of the webinar speakers were also asked this key question by the moderator: “How do you measure success?” This is the point at which webinar content fell to its lowest level: each distinct answer was minimally ‘reasonable’. These companies appeared to have no data readily at hand. Rather, they spoke only in vaguely conceptual terms.
Unable to cite specific results, Jacobson suggested that measuring the success of VR-based training “would not be different from live simulation training: put them in the experience, gather internal data [on performance levels], and determine how to improve these numbers over time:.” According to Jacobson, it’s all about “how well staff performs in live practice and live simulation”. As far as actual results, Jacobson could, at this time, only tout the elimination of overbooking in the hospital’s physical training center.
Moreno also had no specific data to share with the audience, but suggested it may be useful to compare VR training versus standard classroom training in terms of “reaction time, number of errors, and how well participants perform in later live [field] observation.” Davis offered no current results, yet postulated that assessment needed to “move from knowledge transfer to capabilities development and deeper performance”. Clarida agreed, proposing that measurement “should be some sort of performance that can be observed or tracked”, yet failed to add any results from their pilot projects.
Are they measuring success? Not so much. I am alarmed that none of the speakers had any data in hand. I find that this is very typical behavior at innovation’s edge—the fear of evaluation in the educational endeavor. It’s all too common that novices in educational innovation tend to pursue chiefly low-lying fruit. (If you are searching higher up the branches for more aggressive results, see my recent articles, Flying High with Educational VR, VR for PT: Towards Educational Advantage, and How to Score VR for Educational Advantage.) “Shoulda-woulda-coulda” just doesn’t cut it for me. – Len Scrogan
