Contents
Unifying Theme: Closed loop Interventional Medicine
We view the basic paradigm of patient-specific interventional medicine as a closed loop process, consisting of 1) combining specific information about the patient with the physician’s general knowledge to determine the patient’s condition; 2) formulating a plan of action; 3) carrying out this plan; and 4) evaluating the results. Further, the experience gathered over many patients may be combined to improve treatment plans and protocols for future patients. This process has existed since ancient times. Traditionally, all these steps have taken place in the physicians head. The ability of modern computer-based technology to assist humans in processing and acting on complex information will profoundly enhance this process in the 21st Century.
The CiiS Lab focuses broadly on all aspects of the systems and technologies associated with this basic process, including innovative algorithms for image processing and modeling, medical robots, imaging & sensing systems and devices, software, and human-machine cooperative systems. Our basic strategy involves working closely with clinicians, Industry, and other collaborators on system-oriented, problem-driven engineering research.
Note: CiiS and related labs grew out of the NSF Engineering Research Center for Computer-Integrated Surgical Systems and Technology (CISST ERC). The ERC remains active, although the CISST web site has only been sporadically maintained since the $33M NSF seed money grant was spent out in 2009. Volume 1 of the ERC Final Report contains an excellent summary of the ERC progress up to that time.
Projects
Short summaries of the various research activities associated with CiiS are below, together with links to project summary pages. In cases where the project’s “home” is more closely associated with another lab that cooperates with CiiS, then the project page may be found on that lab’s Wiki.
Microsurgical Assistant System
This Bioengineering Research Partnership (BRP) focuses the efforts of highly qualified engineers and scientists from JHU (lead institution), Columbia and CMU, as well as surgeons from the JHU School of Medicine, to overcome human limitations in surgical practice. This project proposes to develop novel core technology and microsurgical tools with unique capability, as well as integrate computer assist systems. The effort will generate a computer assisted human user with enhanced microsurgical ability.
Microsurgical Assistant System Project Page
Eye Robots
The Steady-Hand Eye Robot is a cooperatively-controlled robot assistant designed for retinal microsurgery. Cooperative control allows the surgeon to have full control of the robot, with his hand movements dictating exactly the movements of the robot. The robot can also be a valuable assistant during the procedure, by setting up virtual fixtures to help protect the patient, and by eliminating physiological tremor in the surgeon’s hand during surgery.
Force Sensing Microsurgical Instruments
Retinal microsurgery requires extremely delicate manipulation of retinal tissue. One of the main technical limitations in vitreoretinal surgery is lack of force sensing since the movements required for dissection are below the surgeon’s sensory threshold.
We are developing force sensing instruments to measure very delicate forces exerted on eye tissue. Fiber optic sensors are incorporated into the tool shaft to sense forces distal to the sclera, to avoid the masking effect of forces between the tools and sclerotomy. We have built 2DoF hook and forceps tools with force resolution of 0.25 mN. 3DoF force sensing instruments are under development.
Force Sensing Microsurgical Instruments Project Page
Optical Sensing Instruments
OCT provides very high resolution (micron scale) images of anatomical structures within the tissue. Within Ophthalmology, OCT systems typically perform imaging through microscope optics to provide 2D cross-sectional images (“B-mode”) of the retina. These systems are predominantly used for diagnosis, treatment planning, and in a few cases, for optical biopsy and image guided laser surgery. We are developing intra-ocular instruments that combine surgical function with OCT imaging capability in a very small form factor.
Optical Sensing Microsurgery Instruments Project Page
Multispectral Light Source for Retinal Surgery
It has been shown that retinal surgical procedures often incur phototoxicity trauma of the retina as a result of illuminators used in surgery. In answer to this problem, we have developed a computer-controlled multispectral light source that drastically reduces retinal exposure to toxic white light during surgery.
Multispectral Light Source Project Page
High Dexterity Robotic Manipulators
Snake Robot
The JHU Snake Robot is a telerobotic surgical assistant designed for minimally invasive surgery of the throat and upper airways. It consists of two miniature manipulators with distal dexterity inserted through a single entry port (a laryngoscope). Each dexterous manipulator is capable of motion in four degrees of freedom (dof) and contains a gripping device as the end effector. Three additional dof allow for coarse positioning in the x, y, and z directions, and a shaft rotation dof provides the surgeon the ability to roll the gripper while the snake is bent. Please visit the Snake Robot project page for further information.
Information-Enhanced Surgery
One major focus of research within our Center has focused on exploiting the fact that telesurgical systems such as Intuitive Surgical’s da Vinci robot essentially put a computer between the surgeon and the patient. Examples include the development of virtual fixtures and cooperative manipulation methods, the integration of advanced sensing methods and technology with robotic devices, and augmented reality information overlay.
Image Overlay
The goal of this project is to provide information assistance to a physician placing a biopsy or therapy needle into a patient’s anatomy in an online intraoperative imaging environment. The project started within the CISST ERC at Johns Hopkins, as a collaboration between JHU and Dr. Ken Masamune, who was visiting from the University of Tokyo. It is now a collaborative project between JHU and the Laboratory for Percutaneous Surgery at Queen’s University. The system combines a display and a semi-transparent mirror so arranged that the virtual image of a cross-sectional image on the display is aligned to the corresponding cross-section through the patient’s body. The physician uses his or her natural eye-hand coordination to place the needle on the target. Although the initial development was done with CT scanners, more recently we have been developing an MR-compatible version.
Image Overlay Project Page at Queen’s University
Intraoperative Image-Guided TransOral Robotics Surgery
The goal of this project is the development of stereoscopic video augmentation on the da Vinci S System for intraoperative image-guided base of tongue tumor resection in transOral robotic surgery (TORS). We propose using cone-beam computed tomography (CBCT) to capture intraoperative patient deformation from which critical structures (e.g., the tumor and adjacent arteries), segmented from preoperative CT/MR, can be deformably registered to the intraoperative scene. Augmentation of TORS endoscopic video with surgical planning data defining the target and critical structures offers the potential to improve navigation, spatial orientation, confidence, and tissue resection, especially in the hands of less experienced surgeons.
Intraoperative Image-Guided TransOral Robotics Surgery
Integrated Systems for Image-Guided Interventions
Optimization-Based Virtual Fixtures for Robotic Systems
Our technical approach is based on the constrained optimization framework originally proposed by Funda and Taylor66 and subsequently extended to support a library of virtual fixture primitives. Briefly, we formulate the motion control of the robot as a quadratic optimization problem, in which constraints and optimization criteria are combined from multiple sources. These include: i) joint limits and other robot kinematic constraints; ii) surgeon commands from a master hand controller; iii) real vision or other sensor data; iv) descriptions of desired behavior built up from simple primitives; and v) registered anatomic models of the patient. We solve these problems at interactive rates (typically, from 20 Hz to 200 Hz, depending on problem size) and applied this formulation to teleoperation of complex robots such as snake robots, assistance in tasks such as suturing, implementation of safety regions in neurosurgery, and alignment aids in targeting tasks.
Skullbase Drilling Project
We are developing an image-guided robot system to provide mechanical assistance for skull base drilling, which is performed to gain access for some neurosurgical interventions, such as tumor resection. The motivation for introducing this robot is to improve safety by preventing the surgeon from accidentally damaging critical neurovascular structures during the drilling procedure. This system includes a NeuroMate robot, a StealthStation Navigation system and the Slicer3 visualization software.
Osteoporotic Bone Augmentation
We are developing a software workstation to assist in performing femoral augmentation for osteoporotic patients. The surgical procedure consists of a series of bone cement injections into the femur. The bone augmentation system features preoperative biomechanical planning to determine the most advantageous cement injection site(s) and injection protocol to increase the strength of the femur. During the surgery, we will navigate the injection device and intraoperatively reconstruct virtual models of the injected cement.
Osteoporotic Bone Augmentation
Osteolytic Lesion Curettage with Snake Robot
We have an active collaboration with the Johns Hopkins Applied Physics Laboratory to develop a robotic system for minimally-invasive curettage of osteolytic lesions in bone. One novel aspect of the system is a 6 mm diameter nitinol steerable “snake” end effector with a 4 mm lumen which may be used to deploy a variety of tools into the cavity. Although the initial focusing application is osteolysis associated with wear particles from orthopaedic implants, other potential applications include curettage of other osteolytic lesions such as bone metastases, intra-cardiac applications, and other high-dexterity MIS applications.
Osteolytic Lesion Curettage of Osteolytic Lesions
3D US Snarfer Robot
3D Ultrasound-Based System for Nimble, Autonomous Retrieval of Foreign Bodies from a Beating Heart
Explosions and similar incidents generate fragmented debris such as shrapnel that, via direct penetration or the venous system, can become trapped in a person’s heart and disrupt cardiac function. This project focuses on the development of a minimally invasive surgical system for the retrieval of foreign bodies from a beating heart. A minimally invasive approach can significantly improve the management of cardiac foreign bodies by reducing risk and mortality, improving postoperative recovery, and potentially reducing operating room times associated with conventional open surgery. The system utilizes streaming 3D transesophageal echocardiography (TEE) images to track the foreign body as it travels about a heart chamber, and uses the information to steer a dexterous robotic end effector to capture the target. A system of this nature poses a number of interesting engineering challenges, including the design of an effective retrieval device, development of a small, dexterous, agile robot, and real-time tracking of an erratic projectile using 3D ultrasound, a noisy, low resolution imaging modality with artifacts.
Elastography with LARSnake Robot
The aim of this project is to control the dexterous snake-like robot under ultrasound imaging guidance for ultrasound elastography. The image guidance is through an ultrasonic micro-array attached at the tip of the snake-like robot.In the robotic system developed the snake-like unit is attached to the tip of IBM Laparoscopic Assistant for Robotic Surgery (LARS) system [A] which can also be seen in the Figure on the right.
The main role of the LARSnake system in this project is to generate precise palpation motions along the imaging plane of the ultrasound array. With an ultrasonic micro-array B-mode images of a tissue can be obtained however,if at hand compressed and uncompressed images of the tissue are present, than one can obtain the elastgraphy images(strain images) of the malignant tissue which is the main purpose of this project.
- By: Tutkun Şen Elastography with LARSnake Robot
Robot Assisted Laparoscopic Ultrasound Elastography (RALUSE)
The goal of this project is to facilitate robot-assisted minimally invasive ultrasound elastography (USE) for surgeries involving the daVinci surgical robot. Information from USE images will extend the range of applicable procedures and increase the robot’s capabilities. The daVinci system brings many surgical advantages, including minimal invasiveness, precise and dexterous motion control, and an excellent visual field. These advantages come at the cost of surrendering the surgeon’s sense of feel and touch. Haptic information is significant for procedures such as organ-sparing resections that target removal of hidden tumors, for example. In open surgery, a surgeon may locate an embedded tumor by manual palpation of tissue. Without haptic sensory information, robotic surgery requires other means to locate such tumors. USE imaging is a promising method to alleviate the impact of haptic sensory loss, because it recovers the stiffness properties of imaged tissue. USE peers deep into tissue layers, providing localized stiffness information superior to that of human touch.
Robot Assisted Laparoscopic Ultrasound Elastography (RALUSE)
MRI compatible Robotics
Magnetic Resonance Imaging (MRI) guided robotic approach has been introduced to improve prostate interventions procedure in terms of accuracy, time and ease of operation. However, the robotic systems introduced so far have shortcomings preventing them from clinical use mainly due to insufficient system integration with MRI and exclusion of clinicians from the procedure by taking autonomous approach in robot design. To overcome problems of such robotic systems, a 4-DOF pneumatically actuated robot was developed for transperineal prostate intervention under MRI guidance in a collaborative project by SPL at Harvard Medical School, Johns Hopkins University, AIM Lab at Worcester Polytechnic Institute, and Laboratory for Percutaneous Surgery at Queens University, Canada. At the moment, we are finalizing pre-clinical experiments toward real patient experiment(s) which will be coming up soon. Also, we are granted funding for another 5 years (till 2015) to develop the third version of the robot which not only addresses shortcomings of the previous versions, but also provides the ability to perform teleoperated needle maneuvering under real-time MRI guidance with haptic feedback. Please keep visiting this website for the latest update!
Bio-manipulation
Bio-manipulation tasks find wide applications in transgenic, biomedical and pharmaceutical research, including common biomedical laboratory tasks such as manipulating cells. The objective of this research is to develop and evaluate methods for performing micrometer scale laboratory bio-manipulation tasks. These tasks form basic enabling components of methods for cell manipulation and cellular microinjections in the development of transgenic models, cancer research, fertilization research, cytology, developmental biology and other basic biological sciences research. This research will create infrastructure for performing these bio-manipulation tasks with force and vision augmentation. Our preliminary results promise significant improvement in the performance of these tasks with augmentation. Specifically, we will evaluate comparative performance of tele-operated and direct manipulation methods with and without augmentation in these tasks.
Robotic Laryngeal Surgery
This project aims to design a robotic system to drive a flexible laryngoscope such that it can be used intra-operatively for evaluation and resection of lesions within the upper aerodigestive system (larynx, hypopharynx, oropharynx). The end goal is to produce a robotically controlled distal-chip flexible laryngoscope with a working port such that all movements of the distal tip can be controlled by a single remote control or joystick. Using a robotic system to manipulate the laryngoscope has several distinct advantages over manual manipulation: 1) a robotic manipulator allows for greater accuracy than can be achieved manipulating the scope by hand, 2) since the robot is supporting the weight of the scope, surgeon fatigue is reduced, 3) since the scope can be completely manipulated with one hand, the number of personnel needed for the operation is reduced, 4) since the robot is in between the surgeon and the scope, active software and control features like motion compensation and virtual fixtures can be introduced. This project also has a significant clinical component, since the end goal is a clinically usable system meeting all clinical regulatory requirements.
Laryngeal Surgery Robot Project Page
Medical Imaging and Modeling Projects
Fusion of Images with Prior Data
One common theme in research in our laboratory and elsewhere is combinining prior information about an individual patient or patient populations with new images to produce an improved model of the patient.
Examples may be found throughout our research writeups(e.g., Sparse X-Ray Reconstruction, 3D ACL Tunnel Position Estimation).
Some more examples may be found on the ”Fusion of Images with Prior Data” page.
Sparse X-Ray Reconstruction
The Sparse X-Ray Reconstruction project aims to develop algorithms for reconstructing 3D models of objects from a few X-Ray projection images. The SxMAC algorithm uses multi-view active contour techniques to change the shape of a deformable model until the simulated appearance of the model that would be observed in X-Ray images matches the acquired images. The algorithm is applicable to intra-operative procedures where an object is injected or inserted into a patient, and the surgeon is interested in knowing the shape, orientation, and location of the object relative to the patient. The target application for this algorithm is a surgical procedure where bone cement is injected into a patient’s osteoporotic femur to reinforce the bone. The surgeon is interested in knowing the shape of the cement during the injection process to guide execution of the surgical plan. More Info
– B. C. Lucas, Y. Otake, M. Armand, and R. H. Taylor, “A Multi-view Active Contour Method for Bone Cement Segmentation in C-Arm X-Ray Images”, IEEE Trans.. Medical Imaging, p. in Press (epub date Oct 13), 2011. PMID 21997251.
Geometric Representations for Deformable Models
Deformable models are used in medical imaging for registering, segmenting, and tracking objects. A variety of geometric representations exist for deformable models. There are several imaging applications that execute a series of tasks which each favor a particular geometric representation. There are three major geometric representations: meshes, level sets, and particle systems. Each representation lacks a key feature that hinders its ability to perform one of the three core tasks for deformable models. In order to use the preferred representation for each task, intermediate steps are introduced to transform one representation into another. This leads to additional overhead, loss of information, and a fragmented system. The Spring Level Set (springls) representation unifies meshes, level sets, and particles into a single geometric representation that preserves the strengths of each, enabling simultaneous registration, segmentation, and tracking. In addition, the springls deformation algorithm is an embarrassingly parallel problem that can run at interactive frame rates on the GPU or CPU. More Info
– B. C. Lucas, M. Kazhdan, and R. H. Taylor, “SpringLS: A Deformable Model Representation to provide Interoperability between Meshes and Level Sets”, MICCAI, Toronto, Sept 18-22, 2011.
3D ACL Tunnel Position Estimation
We are developing a 2D-3D registration method using post-operative radiographs to estimate the tunnel positions in Anterior Cruciate Ligament (ACL) reconstructions by registering a patient-specific 3D model of knee with two post-operative X-rays. This could facilitate the precise identification of the tunnel locations as well as the perception of their relationship to the topographical osseous anatomic landmarks. The success in developing and validating the proposed workflow will allow convenient but precise assessment of tunnel positions in patients receiving ACL reconstruction with minimal risk of radiation hazard.
The method may also be used in statistical studies to assess variation in surgical technique and their effects on outcomes.
3D ACL Tunnel Position Estimation
E-science Meets Radiation Oncology
This joint project between the JHU CS Department and the JHU Radiation Oncology Department explores the statistical relationship between anatomic shape and treatment planning in radiation oncology. In one current project, our goal is to use a database of previously treated patients to improve radiation therapy planning for new patients. The key idea is that the geometric relationship between the tumor and surrounding critical anatomic structures has a crucial role in determining a dose distribution that treats the tumor while sparing critical structures. We have developed a novel shape descriptor, the Overlap Volume Histogram (OVH) characterizing the proximity of the treatment volume to critical structures. The OVH is then used to as an index into our patient data base. Retrieved information may be used both for plan quality sanity checking and as a means of speeding up the initial stages of treatment planning.
In closely related work, we explore advanced computational methods for radiation therapy treatment planning.
Follow this link for more information
Software and Systems Infrastructure
The cisst Libraries
The cisst software package is a collection of libraries designed to ease the development of computer assisted intervention systems. One motivation is the development of a Surgical Assistant Workstation (SAW), which is a platform that combines robotics, stereo vision, and intraoperative imaging (e.g., ultrasound) to enhance a surgeon’s capabilities for minimally-invasive surgery (MIS). All software is available under an open source license, which can be found here. The source code can be obtained from our git (GitHub) repository, https://github.com/jhu-cisst/cisst/wiki. The SVN repository contains a public trunk and development branches. The following table describes the cisst libraries and indicates in which state they are.
IEEE 1394 (FireWire)-Based Motor Controller for the Snake Robot
Research in surgical robots often calls for multi-axis controllers and other I/O hardware for interfacing various devices with computers. As the need for dexterity is increased, the hardware and software interfaces required to support additional joints can become cumbersome and impractical. To facilitate prototyping of robots and experimentation with large numbers of axes, it would be beneficial to have controllers that scale well in this regard.
High speed serial buses such as IEEE 1394 (FireWire), and low-latency field programmable gate arrays make it possible to consolidate multiple data streams into a single cable. Contemporary computers running real-time operating systems have the ability to process such dense data streams, thus motivating a centralized processing, distributed I/O control architecture. This is particularly advantageous in education and research environments.
This project involves the design of a real-time (one kilohertz) robot controller inspired by these motivations and capitalizing on accessible yet powerful technologies. The device is developed for the JHU Snake Robot, a novel snake-like manipulator designed for minimally invasive surgery of the throat and upper airways.
FireWire Motor Controller Project Page
Software System Safety for Medical and Surgical Robotics
One of recent trend in robotics is that the workspace of robots is getting closer to or has significant overlap with that of humans. This naturally leads to the issue of safety between robots and humans. In case of medical and surgical robotics – one of representative safety-critical systems – this is a matter of the utmost importance because robots directly operate on, with, or even inside humans. This requires both hardware and software components to be designed with the consideration of safety in all phases of system development.
Building medical and surgical robot systems with safety is, however, not a trivial process because typical computer assisted intervention applications use different sets of devices such as haptic interfaces, tracking systems, imaging systems, robot controllers, and other devices. Furthermore, this increases complexity of a system, bringing the issue of scalability and maintainability as well. Thus, systematic ways to deal with these issues is necessary.
Fault tolerance is known as one of effective ways to achieve software system safety. As a first step towards software system safety, we develop systematic ways to make component-based software systems fault tolerant and thus to provide medical and surgical robot applications with a basis for safety. Currently, we use the cisst libraries and Surgical Assistant Workstation (SAW) as proof-of-concept platforms.
Task Analysis
Sensor-based Activity Recognition in an Intensive Care Unit
Process inefficiencies and underspecified physician orders and protocols in an Intensive Care Unit (ICU) can induce unnecessary workflow problems for nurses and cause sub-optimal conditions for the patient. During a patient’s stay there are many hundreds of small tasks that need to be completed, often in a specific sequence, and coordinated in time by a team of up to a hundred staff members. These include actions like giving medication, emptying chest tubes, and documenting vital signs. We are working on developing activity recognition techniques for logging the patient-staff interactions to improve workflow and potentially increase patient safety. An Xbox Kinect and other sensors are being used to monitor the ICU over extended periods of time and will be critical in identifying the tasks being performed.