Recently I wrote about Augmented Reality (AR) and Virtual Reality (VR) for mainstream applications. One of the mainstream applications discussed in the Harvard Business Review was the use of VR by surgeons as an aid in surgery, including both planning and executing the operations.
The use of the Vista Stereosite HMD during endoscopic surgery in the 1999 – 2002 time frame. (Credit: University Medical Center, Nijmegen, The Netherlands)
Any system or device to be used in the US must be approved by the FDA, in a process called 510(K) Clearance, before clinical use and that includes AR head mounted displays (HMDs). The earliest approval of a HMD for medical use I found was from 1996 for a Vista Medical Technologies system. The FDA 510(K) approval form (#K961800) says:
“The Head Mounted Display (HMD) System is a binocular field display, occupying a large portion of the normal straight ahead field of view. This display is intended to be used as a component of a visualization system for use in endoscopic surgery. Traditional video-endoscopic surgery images are displayed on a TV monitor. This requires the surgeon to maintain contact with the traditional monitor and remain in position with it at all times. Having to turn away from the monitor causes the surgeon to lose sight of what is being accessed. The HMD system enables the surgeon to see the surgical site as well as the general operating field with the lateral and 45 degree look down vision, without having to look away, thus providing the means to present the endoscopic picture in a more natural and acceptable look down location with respect to the patient. This also provides a safe back up, the physician can resort to either direct visualization or utilize the back up 2D video monitor. The HMD takes a standard signal and displays it on two small electronic image display devices called liquid crystal display (LCD). It can be connected to any standard video generating equipment.
“The HMD consists of an image generating array, eyepiece, and electronics. The image generating array is an active matrix liquid crystal display (LCD). These elements are mounted on a lightweight and compact headgear which fits comfortably on the user’s head. The remaining electronics are housed separately in a controller station. The HMD communicates with the controller by a lightweight, durable, flexible cable. The HMD can be driven by both monocular and stereo endoscopes.
“The HMD Controller will accept NTSC, PAL with CVBS, or S Video (Y,C) video input. Interfaces to external and remote data sets, power supplies and drivers for the LCD and backlight are included in the controller. A maximum of four (4) HMDs and two (2) standard video monitors can be driven simultaneously with the single controller.”
Other than the NTSC and other analog inputs and the use of a LCD rather than a OLED microdisplay, this description could match a modern HMD. E. J. van Lindert, M.D. Ph.D. and his colleagues in the Neurosurgical Department, University Medical Center, Nijmegen, The Netherlands published a clinical study on the Vista Medical Technologies Stereosite Visualization & Information System HMD in 2004. The clinical study was based on 269 endoscopic procedures performed between July 1999 and June 2002. The article concludes “In only one case was it necessary to abandon use of the HMD due to inferior visualization; in all other cases visualization by the HMD was thought to be sufficient. Fatigue of the surgeon due to wearing the helmet did not occur. All surgeons had the impression that visual strain was decreased in comparison to looking at a monitor from a distance. The working position was considered to be more comfortable when wearing the HMD, and eye-hand coordination was improved.” Note that in this case, the HMD was showing real-time images from cameras on the endoscope inside the body, not computer-generated images.
ConMed, Vista Medical Systems successor, uses conventional 3D technology for its surgical visualization systems in 2018 (Credit: ConMed)
Vista Medical Systems was bought by Viking Systems in 2002 and Viking was in turn bought by ConMed in 2012. ConMed, somewhat disappointingly, does not offer a HMD for use with its stereoscopic 3DHD visualization system for use in endoscopic surgery. Instead, they offer a conventional 3D display, a 24” monitor from Sony and a 32” monitor from Panasonic. The ConMed video on the visualization system says it has HDR and calibrated color and the company also offers 4K monitors as an option.
Novarad Banner for the use of a HMD for medical imaging. The system uses gesture recognition to allow the doctor to select menu items without touching anything and potentially contaminating his sterile gloved hands. (Credit: Novarad)
Even if ConMed does not offer HMDs for medical visualization, other companies do. For example, Novarad says its OpenSight AR System is the first solution for the Microsoft HoloLens mixed reality system that is 510(k) cleared by the FDA for medical use. However, the system is not cleared for use during the operation itself, but only for planning the operation beforehand and other similar tasks. What the OpenSight system does is overlay computer generated virtual 3D images, often sourced from a CT scanner, on the actual patient so the surgeon can see the relationship between internal organs and the patient’s external features.
Microsoft HoloLens Mixed Reality HMD (credit: Microsoft)
The OpenSight system with a HMD is said to provide the following benefits:
- Anatomically Correct: OpenSight projects rendered 3D images onto the patient with high accuracy. This allows users to pinpoint areas for training, identification or for further analysis. This rendering of virtual images to match real objects for surgical applications is described in US 9,892,564, assigned to Novarad. The company also has three other issued patents.
- Better Pre-Op Planning: With the ability to see the patient’s anatomy surgeons can strategize and plan more easily before making any incisions.
- Enhances Scholarly and Medical Research: OpenSight will enhance medical research by providing researchers with an unparalleled view of the subjects anatomy without the need for surgical incisions or other invasive procedures.
- Teaching Tool: Medical students and professors can use OpenSight as a teaching tool that will help students visualize and understand patients’ internal anatomy. OpenSight can be used to either replace cadavers or overlay CT scan data on cadavers enabling residents to hone their craft with guidance from AR.
When used with the Microsoft HoloLens, the OpenSight visualization system can overlay 3D virtual images on and inside a patient. (Credit: Novarad)
While Novarad uses an off-the-shelf HMD in its visualization system, others are working on developing dedicated HMDs for medical applications. For example, Kopin and Endopodium have formed a joint venture, HMD/MD, to produce a state-of-the-art medical HMD using Kopin OLED microdisplays. According to the announcement from Kopin, “Endopodium was founded by the team which developed and commercialized the Vista Medical HMD, the first medical grade HMD cleared for marketing by the FDA.”
Endopodium Surgical AR 3D HMD Design, to be developed with Kopin. (Credit: Endopodium)
In commenting on the agreement, John Lyon, co-founder of Endopodium, said, “Our earlier work [at Vista] was limited by the size and performance level of the displays available. Now, with Kopin’s leading OLED technology, we can create wearable systems which are ergonomic, lightweight and high resolution to meet the critical demands of today’s medical and surgical applications.”
Under the terms of the agreement, Kopin will provide both intellectual property and OLED displays, as well as technical expertise in configuring state-of-the-art wearable systems for mission-critical applications. HMD/MD plans to establish strategic distribution partnerships with medical technology companies that wish to include high resolution wearable displays in their proprietary visualization systems. –Matthew Brennesholtz
