[Back to projects]

Telerobotics

Some of my previous work at the NVL was for the Integrated Remote Neurosurgical System (IRNS), an experimental system designed to allow a neurosurgeon to remotely operate a surgical microscope.

My job was to design the user interface for the system, while the other folks on the team were doing the down-and-dirty robotics work.

Graphical User Interface

Below is what the GUI looked like. Not much has changed since then except for the addition of a few buttons to clear the whiteboard annotations on the video windows, and some extra markers on the MRI images to indicate current trajectory and planned trajectory. A switch was also added to kill remote control should an emergency arise. I pasted some pictures where the live video should be since my screen capture utility doesn't handle the extra frame buffer on the video card.

Three Views of the IRNS GUI

Fig 1. Operating Room GUI
Fig 2. Remote GUI	Fig 3. Remote GUI (snapshot facility shown)

Layouts

First, you'll notice that the O/R and remote GUI's are nearly identical. The only big difference is that the volume rendering is shown in a different place. I wanted to have it in a larger and continually visible window for the O/R surgeons, because when provided with such tools, they use them quite frequently. For example, I observed several procedures where continuous images of the robot's trajectory were available on a computer screen. Given this facility, the surgeons and residents continually referred back to it as an additional safety check.

I was never happy with the layouts of the interfaces, so I relied on the comments of others to fix them. Some work has already been done by previous members of the lab, and that influence is reflected above. Also, we made extensive use of touchscreens at the NVL, and that is why all the buttons you see in the GUIs are rather bloated. Surgeons and residents periodically demo the system and provide critique, and I conducted several observations of neurosurgical procedures at the UVa Hospital.

The local GUI provides a view from the microscope (the black box in the top left of Fig. 1). This window receives annotations from the remote surgeon and allows someone in the O/R to annotate the image as well.

The remote GUI, on the other hand, contains two camera views - a view from the microscope (top left of Fig 2 and 3) and a room view of the O/R (top right of Fig 2 and 3). The O/R camera can be moved by pointing and clicking on the image (the camera will center the clicked point). A box can be dragged out on the O/R view window to center and zoom to a particular area.

The remote surgeon can pan the robot using the arrow buttons surrounding the window in Figs 2 and 3. A whiteboard facility is provided on the microscope view whereby the surgeon can annotate the image and have his notes appear on the GUI in the O/R (Fig 1).

Surgical planning facilities (some of which were developed previously here at the NVL) are shown in the lower portions of Fig 1 and 3. The surgeon can access patient MRI data and interactively scan through it. The three orthogonal views (axial, coronal, sagittal) are provided as well as an oblique view that can follow the surgical plan or the robot trajectory. A volume rendered image is also provided that displays the planned trajectory of the surgery. It rotates along with the scope to provide a simulated view of the patient which can be compared with the live video.
It is also possible for either the remote or local team to specify their own trajectory and move the robot to align with it. This is particularly useful for high-latency situations where real-time interaction may be difficult. The user can see the future location of the robot and simply click a button to move it there. An interesting paper from the University of Maryland addressed this issue in the design of a telepathology workstation.

The final component (at this point) is the snapshot facility, which is provided in both the local and remote GUI's but is only shown here in Fig. 3. The surgeon can take stills of the microscope view by clicking the " Take Snapshot " button. The software also saves any annotations made by the surgeon and allows him/her to toggle whether they are displayed. Snapshots are whiteboards themselves, so annotations can be made to them at any time by either the local or remote staff. The state of the robot (joint angles) is also stored to allow the surgeon to return to the position of a previous snapshot. A small menu of thumbnail images is provided to the left of the snapshot window to give the surgeon quick access to old snapshots.
I added an audio annotation facility to let anyone record something to help them remember what they were doing when they took a particular snapshot. A small set of voice commands are supported for those who have experience on the actual O/R robot (the Zeiss MKM), which has voice recognition capabilities.

Robot Manipulator

After quite a bit of debate, we settled on the use of a small desktop robot arm, the Puma 260, as our means of controlling the robotic microscope remotely. Use of a robot arm provides configurability in the number of degrees-of-freedom given to the user, the possibility of force feedback, and the capability of dexterity enhancement. In addition, we can simulate the current (local) interface on the robotic microscope for comparison.
The Puma is currently equipped with an identical hand controller and force sensor to that of the Zeiss MKM. This makes it possible to closely mimic the interaction that surgeons would have if they were locally manipulating the microscope.
I have concerns that the lower bandwidth of the forearm as compared to the fingers and wrist may detract from the remote surgeon's performance in controlling the robot. I would like see a comparison test with a stylus-based interface to the robot in the future.

In any case, once development is completed, the Puma can be tested in a number of different configurations (with/without force feedback, simulating the current interface, etc.) to determine the most effective means of control.

Our registration system uses a Flashpoint optical tracking system and patient/image registration software to allow us to move the robot with respect to the patient or the patient's image data.

References

• Graves, S., Tullio, J., Shi, M., Downs, J.H., An Integrated Remote Neurosurgical System, First International Joint Conference CVRMed - MRCAS '97. [PDF]

• Buxton, W. Telepresence: Integrating Shared Task and Person Space, Proceedings of Graphics Interface '92 [HTML]
• Carr, D., Hasegawa, H., Lemmon, D., Plaisant, C. The Effects of Time Delays on a Telepathology User Interface, Proc. of the 16th Annual Symposium on Computer Applications in Medical Care (SCAMC), Baltimore, MD, 1992.
• Milgram, P., Rastogi, A., Grodski, J. Telerobotic Control Using Augmented Reality,Proceedings of 4th IEEE International Workshop on Robot and Human Communication, 1995.
• Nardi, B., Schwarz, H., Kuchinsky, A., Leichner, R., Turning Away From Talking Heads: The Use of Video-as-Data in Neurosurgery, Proceedings of InterCHI '93.
• Salcudean, S.E., Yan, J. Towards a Force-reflecting Motion-scaling System for Microsurgery, Proceedings of IEEE Int'l Conference on Robotics and Automation, 1995.
• Schenker, P., Barlow, E., Boswell, C. Development of a Telemanipulator for Dexterity-Enhanced Microsurgery, Proceedings of Medical Robotics and Computer-Assisted Surgery '95.
• Taylor, R., Funda, J., Eldridge, B., Gomory, S. A Telerobotic Assistant for Laporoscopic Surgery, IEEE Engineering in Medicine and Biology, May/June 1995.
• Yamaashi, K., Cooperstock, J., Narine, T., Buxton, W. Beating the Limitations of Camera-Monitor Mediated Telepresence with Extra Eyes, Proceedings ACM CHI '96 Conference on Human Factors in Computing Systems.