AR/VR innovations in surgery and healthcare

Virtual Reality in the healthcare market is expected to reach $3,8bn by 2020, demand is far from slow down in healthcare system, and day by day new company and research centers are offering AR/VR based solutions for the healthcare market. According Goldman Sachs Global Investment Research estimations becomes $5,1bn by 2025.

 

 

Current and future healthcare solutions include a wide range of applications, from educational purposes, training of future physicians, guided surgery, tele-medicine, physical therapy and treatments of phobias and mental diseases phobias.

Focusing in surgery, involving not just the training of surgeons, VR visualizers for anatomy studies, pre-planning of surgeries, AR/VR guided and assistant at the operation theater, robotic and tele-surgery.

According to recent studies, 18% of U.S. physicians in 2016 had used VR for professional purposes, and 3/5 are interested in using VR for medical training and assistance. More than a half were interested in the use of VR in new medical treatments and conditions. The healthcare sector is receptive to this revolutionary technology, and surgery is one of the most active fields.

 

AR/VR applications in healthcare

Most of us are familiar with impressive videos of VR immersive 3D videos showing human body anatomy but applications in healthcare of this amazing technology are pushing several cases that includes a wide range of medical disciplines.

AR/VR can revolutionize future medicine providing advanced applications that covers different fields of applicatio, involving:

• Educational projects
• Visualization of 3D data from diagnosis (CT, MRI and ultrasonic imaging)
• Teaching and Training of future physicians
• Radiotherapy treatments planning
• Computer assisted surgery in all their stages from planning using 3D diagnoses up to the assistant during surgery.
• Plastic surgery and dentistry (from previous visualization of the results up to assistant during operation)
• Orthopedic surgery (from planning/prosthesis analysis to guided assistance during operation)
• Telemedicine (from diagnosis and remote assistance up to the most impressive tele/robotic surgery)
• Rehabilitation and physical therapy
• Phobias and mental diseases treatments

Advantages of AR/VR technology in all of these scenarios are numerous and some of them are:

• Time and cost reduction in the training of future doctors and surgeons.
• Surgeon skill level analysis without patient injuries.
• Reduction of cadaver needs in physician training.
• Safe environment for both doctors and patients.
• Real-time feedback of the data during surgery planning.
• The medical assistance in remote locations.
• Most comprehensive diagnosis for patients.
• Immersive treatment of difficult diseases (phobias, etc.).
• Time reduction in rehabilitation and physical therapy.

 

A great discovery: VR/AR for medical data visualization

Several of 3D images are managed by surgeons and other medical disciplines including Computer Tomography (CT), Magnetic Resonance Imaging (MRI) and ultrasound images.

Surgeons often have to build the mental 3D model of a complex biological structure (brain, liver, spine, etc), and its demonstrated that a better spatial, distance and volume estimation is achieved by surgeons using VR environment rather on a 2D desktop based application. [5]

 

 

Qualitative results from surgeon experience using VR environments fitted with position trackers and head-tracked visualization devices show their preference for VR environments.

Awesome hardware and software progress allow nowadays an excellent method for visualize the anatomic model of the human body, or the specific 3D anatomic data of the patient (internal organs, neuronal circulatory system, bones, etc.). An exhaustive planning of the surgical operation to increase the success rate of the surgery.

 

 

And for a patient, after image recording procedure like MRI the doctor could check the position inside the patient of the organ allowing an immediate feedback that no other technology provide.

 

 

Surgical training and surgical simulators, VR “upper hand”

VR simulation as training tool has been deeply integrated nowadays in several medical residency programs. Several studies demonstrate improvements in surgical residents trained on VR simulators. Trained residents were almost 30% faster in surgical dissections and made 1/6 of the errors of their non-simulator trained counterparts. [1][2]

There are two main groups of surgeries: Open Surgery and Minimally Invasive Surgery (MIS). First one requires for simulation a more complex computation due to less restrictive scenario, and its flexibility in the interaction between surgeon and patient. Second restrict the procedure and interaction, but requires that surgeon get enough skill by training, and in this point is where the simulators, and most recently VR simulators reach a big success in terms of safety and cost effectiveness.

 

 

Several surgery domains and clinical applications are currently using VR/AR surgical planning or surgical training tools, including:

• Laparoscopy (VR training/simulators, AR assistance surgery)
• Ocular surgery (VR training/simulators)
• Liver surgery (VR training/simulators, AR assistance surgery)
• Neurosurgery (VR training/simulators, AR assistance surgery)
• Spine surgery (VR training/simulators, AR assistance surgery)
• Maxillofacial surgery (VR training/simulators, AR assistance surgery)
• Breast surgery (VR training/simulators, AR assistance surgery)
• Plastic surgery (VR training/simulators)
• Orthopedic surgery (VR training/simulators, AR assistance surgery)

 

 

Main advantages of the use of VR/AR simulators and assistance during surgery:

• Improve learning curve and reduce the errors during the learning stage of future surgeons
• There is a shortage of cadavers for medical research
• More realistic imitation of a human body inside VR environment
• Helps to visualize complex anatomic structures (for example: neurosurgery)
• Increases speed and accuracy of surgical procedures
• Reduces patients trauma and risk
• Assist surgeons during surgical procedures

Regarding first point the retention level a year after VR training session can be as much as 80%, compared to 20% retention after a week with traditional training.

Initially this kind of simulators were very expensive, but progress of VR/AR industry is providing this technology to much more users. For example Osso VR provide cost effective VR orthopedic surgery simulation that teaches surgeons how to use medical devices in the operating theater, guiding through different steps of the surgical procedure and improving their skills without putting patients at risk.

 

 

Before diagnosis and/or surgery doctors and specialists peer at MRI and CT scans. This amount of data is difficult to visualize as a 3D image, companies like Bioflight VR build a solution that takes these data to display in VR/AR environment. Using this kind of platforms medical professionals can analyze, interact and identify conditions, viewing details that 2D static images rarely reveal.

 

VR/AR surgical simulators, a brief approach

VR/AR surgical simulators should have following characteristics:

• Visual fidelity
• Interactivity
• Force feedback

Visual fidelity means that need to generate convincing visual information for the surgeon. Nowadays AR/VR hardware and software progress achieve this target.

Interactivity to react to surgeon visualization areas and position change, and manage (mainly in AR systems) collision between real and virtual objects.

Force feedback typically with the use of haptic devices. Allow the surgeon react feeling the applied force and movement applied during the simulation.

 

3D tracking for AR/VR applications

Tracking systems measures the three dimensional position (x,y,z) and the orientation (defined by three angles: pitch, roll, yaw) of an object with respect a reference coordinate systems. Combining this information a full description of the six degrees of Freedom (6DoF) allow full spacial and motion tracking of the object.

Five categories are used to rank a tracking device:

• Accuracy (Degree of with the measured data match the true values)
• Robustness (respect corrupted or missing measurement)
• Range and degrees of Freedom.
• Update rate. The speed of the system is going to be determined by the update rate (data/second) and latency (delay between the capture of the object pose and the display in the VR/AR world)
• Hardware (how complex is the system and also its size and weight)

Tracking technologies include magnetic, acoustic, mechanical inertial and optical trackers.

Magnetic trackers are based on generated magnetic field and discrete sensors (three coils placed perpendicular to each other defining a spacial reference). Measuring the field variation of the sensor coils allow a tracking sensor with 6DoF.

AC and DC magnetic systems could be used and second one is more robust against distortion fields created by metallic elements (usually in surgical operation theater there are several metallic elements that can affect the magnetic field).

Main advantages of magnetic trackers is low latency and no line of sight is needed. Another advantage is the small size that could be achieved using small coil sensors. And one of the most important is that several position trackers should be managed using just one magnetic field emitter.

Acoustic trackers are based on ultrasonic waves to determine the position and orientation of the object. Its main disadvantages are low response rate and that can be affected by the temperature and humidity. Also suffer an accumulative error due to the phase difference measured of the acoustic signal.

Mechanical trackers include sophisticated encoders or potentiometers, but requires complex mechanical section and joints so have constraints of movement.

Inertial sensors are based on gyroscopes and accelerometers, as position and movement is achieved from a double integration of acceleration, that produces an accumulative error over time.

Optical trackers provide maximum accuracy, but has the main disadvantage of need a line of sight between the led or tracked by a camera object.

 

 

In other cases 3D tracking is needed to control the position of elements of the human body during the surgical procedure like in the Computer-Assisted Orthopedic Surgery (CAOS). This discipline that we can name Intra-Operative Position Sensing are usually made by optical systems, that are very awkward, and always need a line of sight. Several researches are been made to replace them with electromagnetic tracking systems.

 

Augmented reality, enhancing standard surgical procedures

In contrast with Virtual Reality (VR) systems that immerse the surgeon in immersive full computer generated world, Augmented Reality (AR) generates a composite view of virtual and real objects to the surgeon. Target of added virtual objects is to enhance or add information displayed to the surgeon, in the operation theater or before, during the planning stage of the surgery.

These differences means that 1) in AR need to create just the virtual data, but should mesh them properly with real user view and 2) Latency and display lag of the system should be minimal, the overlaid virtual objects movement should be noticed in correct position. Electromagnetic 3D tracking sensors that can help to this purpose.

 

This AR technology is been tested in surgical procedures for complex structures like neurosurgery, maxillofacial and spine surgery. In all of them, a comprehensive 3D visualization of the body structures of the patient is needed by the surgeon.

 

 

Future and beyond: Tele and remote surgery

One of the biggest challenge for VR in surgery is to face telesurgery, procedures where surgeons and patient are at different locations. Physicians should have a VR produced copy of a remote environment including the patient at their physical location. Combination of advanced technologies for precisely tracking of surgeon movement, a realistic and real time VR environment display for the physicians and codification and telecomunication techniques to transmit the data between two remote locations is needed to accomplish successful these applications.

Main advantages of this promising technology includes [3]:

• In some cases, injured in accidents require immediate surgery in the scene of accident to have better chances of survival
• Patients could be operated by an expertise surgeon who can be located everywhere
• Patients too ill or injured to be transported to a hospital may be operated remotely

Robotic surgery guided by a remote surgeon is a field that is not new, but has boosted recently with the addition of VR technology.

 

AR/VR treatments other than surgery

Surgery procedures have been benefit from VR/AR technology, but there are many other medical treatments that can benefit from this revolutionary technology:

Palliative treatments focused in provide a better condition of patients with painful treatments like chemotherapy or for patients that suffer anxiety during other medical procedures.

 

Phobias or treatments for mental diseases like autism. Several clinics are applying VR immersive experience to help patients get over fear of heights, flight, animals, etc. Regarding autism, there are several projects to allow that kids with autism learn some basic social skills.

Physical therapy or rehabilitation after an accident or orthopedic intervention surgery requires a long time with repetitive routines that can be more pleasant with inside an immersive VR environment, besides the fact that VR 3D tracking of movements allow correct them and evaluate the progress of the treatment.

 

References:

[1] Grantcharov, T.P., Kristiansen, V.B., Bendix, J, et Al, Randomized clinical trial of virtual reality simulation for laparoscopic skills training. BJS 2004:91 (2): 146-150

[2] Seymour NE, Gallagher AG, Roman SA, et al, Virtual reality training improves operating room performance. Ann Surg. 2002 Oct; 203 (4): 458-63

[3] Milgram P, Zhai S, et al, Applications of augmented reality for human-robot communication. In International conference on Intelligent Robotics and Systems, pp. 1467-1472, July 1993

[4] Rolland JP, Baillot Y, et al, A Survey of Tracking Technology for Virtual Environments, chapter 3, pages 67-112. Fundamentals of Wearable Computers and Augmented Reality, 2001

[5] Reitinger B, Schmalstieg D, et al, Spatial Analysis Tools for Virtual Reality-based Surgical Planning. IEEE Symposium on 3D User Interfaces, March 2006, P:25-26

[6] Litvack ZN, Lovel DA, State of the Art: Use of Simulation in Neurosurgery Residency Programs, AANS Abstract, 2010

[7] Simon DA, Intra-Operative Position sensing and Tracking Devices. Proceedings of the first joint CVRMed/MRCAS conference, 1997, pp 62-64.

[8] Pandya A, Auner G, Simultaneous Augmented and Virtual Reality for Surgical Navigation. Annual Conference of the North American Fuzzy Information Processing Society – NAFIPS 2005, pp: 429-435

[9] Buckley CE, Nugent E, Ryan D, et al, Virtual Reality – A new era in surgical training INTECH Open access publisher 2012

[10] Solomon O, Kósa G et al, Enhancing endoscopic image perception using a magnetic localization System, Computer Assisted Surgery, 2006