Abstract Details

Towards Diagnostic Applications Of Virtual/Mixed Reality To Brain Scans

Abstract. The human brain is among the least understood organs in our body, containing billions of nerve cells (neurons) and their connecting fibres (axons and dendrons). A bundle of axons make up a nerve tract that transmits electrical signals to other neurons, muscles, and glands throughout the human body. Over the last few years, clinical visualisation techniques of brain nerve fibres have been developed to enhance the performance of medical diagnostics and neurosurgery. One of them is 2D MRI brain scans (raster) while another is the use of 2.5D visualisation software, such as ExploreDTI, to connect up nerve tract pathways and visualize off-axis viewpoints of tract trajectories (vector).

Recent advances in 3DVR computer gaming engines, such as Unity3D, and 3D visualisation technologies like the Oculus Quest and Microsoft Hololens provide plenty of opportunities for medical visualisation applications. Applying Virtual (VR), Augmented (AR), or Mixed Reality (MR) technologies to the medical sphere shows enormous potential. Studies on the preoperative use of VR/MR applications in complex surgery have made the communication between doctor-patient and even doctor-doctor simpler, more convenient, more intuitive, and more accurate than in the past. For example, 3DVR technology can facilitate a large variety of risk-free medical student training, assist neurosurgeons with preoperative planning/visualisation, and with the introduction of digital avatars in this project, allows for the possibility of real-time collaborative visualisation/interaction of 3DVR brain scans between networked medical practitioners all over the world.

The Digital Games Programme in the School of Media at Technological University Dublin (TUDublin), in collaboration with Trinity College Dublin Institute of Neuroscience (TCIN), have recently developed an interactive 3D brain fibre model in a Unity3D gaming environment by employing brain tract data and neuroimaging technologies. An Oculus Rift headset was used in this project to realise the first ever virtual dissection of the fornix in vivo in a highly interactive full 3D environment.

The aim of our project is to progress existing standalone 2.5D brain tract tools to networked 3DVR for collaborative visualisation and analysis of brain structures. Our research applies VR/MR technology to traditional diagnostic approaches and adds advanced functionality that increases the speed and accuracy of analysing brain nerve tract structures in MRI scans.

To generate the 3D virtual brain model, translating MRI scan data into the Unity3D rendering engine is an essential first step. Selection and manipulation functionality is enabled in our application using virtual hand tracking tools. In comparison to previous methods, adding positional data into MRI images makes this application more accurate. In addition, we propose networking functionality to this application allowing for communication between neurosurgeon avatars from remote areas, within a full 3D virtual environment.

Back to list of abstracts