Université d'Auvergne Clermont1 | CNRS


Axis 2: Computer Vision Aided Laparosurgical Gesture


Laparosurgery has inherent issues in spite of the highly sophisticated tools it uses and high quality images being obtained. As we mentioned in the introduction of the document, the pathological model we use is the one of endometriosis, and in particular of infiltrating endometriosis. The 2D visualization the laparoscopy gives is insufficient to find, visualize and optimally treat the wholeness of lesions. Indeed, owing to the shrinking and deformations it induces, endometriosis considerably changes the anatomical proportions, which makes surgery difficult. Moreover the shape of infiltrating lesions has a high variability and can be only weakly predicted from the images available to the surgeon when the decision of which tissues to resect during exeresis has to be taken.

Clinical and technological objectives

The main clinical objective of this axis is to create tools for perioperative aide and preoperative image processing to improve diagnostic, through the two following points:

  • Perioperative use of multimodal preoperative images. Echography and MRI are two modalities that allow one to make a non invasive diagnosis, and if necessary, to characterize the lesions. These two modalities are complementary. They allow one to make an approximation of the endometriosis’ boundaries, but merging these two types of images should improve the quality of preoperative diagnostic. It is difficult for the surgeon to transpose this preoperative information during surgery. We propose to digitally register them on the laparoscopic view in real-time. That will allow the surgeon to improve safety in exeresis, to make it more complete, reliable and fast. The structures will be marked by overlaying the laparoscopic view and using transparency. Beyond pathological structures, our technique will allow us to mark and emphasis sensitive structures, such the ureters, the vascular network and the pelvis’ nerves.
  • Perioperative 3D vision. The absence of 3D vision is one of the limiting factors in laparosurgery. The current laparoscopes only sense a 2D projection of the abdominal cavity, and do not let the surgeon to perceive depth and distances. We are developing a software that will reconstruct depth from a single image from a standard laparoscope, and will display it on a 3D screen. The surgeon will then be able to make some gestures more easily. Preoperative images such as 3D IRM will be used to predict hidden parts or else to locate vulnerable neighboring organs like the ureters, invisible without dissection.

Technical issues.

The above-mentioned clinical goals require the use of existing tools and the development of new tools that lead to the following technical problems:

  • Registration of 2D echography to 3D MRI. Multimodal registration is an old problem in medical image processing. However, the particular 2D echography to 3D MRI registration problem is difficult and still open.
  • Visual understanding of preoperative images – anatomy recognition, lesion detection and characterisation. Like in axis 1, it is here necessary to build prototypes to model a priori knowledge on the shape, anatomy and appearance of the organs. Statistical models of the anatomy (intra- and inter-patients) combined with morphological data will be developed.
  • Calibration and tracking of the laparoscope and other surgical tools. So as to know the internal and external parameters of the laparoscope, and its shape, we are creating a computer vision based tracking system and a geometric and photometric (optics’, light source and endoscope’s shape geometry) calibration procedure. There already exist such tracking tools. They are however based on markers attached to the structures to be tracked ; makerless tracking is an open research problem.
  • Real-time registration of preoperative images to the laparoscopic video stream. One here has to register a 2D or 3D preoperative iamge with the 2D laparoscopic view. It is a difficult problem for surgery modifies the position of organs in the pelvis during the intervention. We use the fact that the laparoscope’s pose is tracked and that the pelvis is a rigid structure, and estimate only the deformable part of the registration.
  • Real-time 3D reconstruction. Laparoscopic images are difficult to process for classical 3D reconstruction algorithms since these images typically have numerous specular reflexions, and the environment is deformable. So as to overcome these issues, we draw on various image cues, namely shading and motion. We will use recent 3D reconstruction techniques for deformable environments that we partly contributed over the last few years.