Université d'Auvergne Clermont1 | CNRS

.

FLEXABLE (Deformable Multiple-View Geometry and 3D Reconstruction, with Application to Minimally Invasive Surgery)

5 years funded project – ERC Starting Consolidator Grant – GA307483 - on-going project

Partner's funding: 1.481.294,00€

Description

Project FLEXABLE lies in the scientific field of Computer Vision which studies the computational interpretation of images. In particular, 3D Computer Vision seeks to recover depth or the 3D shape of the observed environment from images. One of the most successful and mature techniques in 3D Computer Vision is Shape-from-Motion which is based on the well-established theory of Multiple-View Geometry. This uses multiple images and assumes that the observed environment is rigid.

The world is however made of objects which move and undergo deformations rather than remaining rigid. Researchers have tried to extend Shape-from-Motion to a deformable environment for about a decade, yet with only very limited success to date. We believe that there are two main reasons for this. Firstly there is still a lack of a solid underlying theory for Deformable Shape-from-Motion. Fundamental questions, such as what kinds of deformation can facilitate unambiguous 3D reconstruction, are not yet answered. Secondly practical solutions have not yet come about because the Motion cue has been mostly used on its own. For accurate and dense 3D shape results, it must be combined with other visual cues, since it is certainly weaker in the deformable than in the rigid case. Furthermore, strong priors are needed, requiring one to bridge the gap with object recognition.

This project focuses on developing these two key areas and in doing so will include three main lines of research: theory, its computational implementation, and its real-world application. Deformable Multiple-View Geometry will generalize the existing rigid theory to the deformable case and will provide researchers with a rigorous mathematical framework that underpins the use of Motion as a proper visual cue for Deformable 3D Reconstruction. Our theory will require us to introduce new mathematical tools from differentiable projective manifolds to Computer Vision. Our implementation will study and develop new computational means for solving the difficult inference problems formulated in our theory. Because we strongly believe that Motion is not sufficient for solving the general Deformable 3D Reconstruction problem alone, it will be combined with other visual cues including Shading. Finally, we will develop cutting-edge applications of our framework specific to Minimally Invasive Surgery, with a focus in interventional laparoscopy, for which there is a very high need for 3D Computer Vision.

Project FLEXABLE is a Seventh Framework Programme funded by the European Research Council (ERC). The project is conducted at ISIT within the ALCoV group.

Contact: flexable@udamail.fr

 

 

Members:

 

 

Post Doc - Since April 20, 2015 - Research topic: Shape-from-Template Meets Object Recognition

 

Post Doc - Since April 18, 2016 - Research topic: Benchmarking SfT / SfT and the environment

 

Post Doc - Since July 4, 2016 - Research topic: On the Well-posedness of SfT in the Isometric and Non-Isometric Cases, with Common Types of Inputs

 

PhD Student  - Since Septembre 1, 2014 – Resarch topic: 3D Modelling of deformable materials

Master's Intern - February – June, 2014 – Master in Computer Vision, Université de Bourgogne – Resarch topic: Modeling Multiple Images of a Deformable Surface Using Differential Projective Geometry

 

PhD Student - Since Septembre 1, 2014 – Resarch topic: 3D Monocular Reconstruction in an Uncontrolled Environment and Medical Applications

 

Intern - January - July, 2017 - Research topic: Does elastography support 3D reconstruction?
 

 

Collaborators:

MD University Hospital (CHU) and Université d’Auvergne

 

MD University Hospital (CHU) and Université d’Auvergne

 

Associate Professor at Université d’Auvergne

 

Researcher at Université d’Auvergne

 

Colleague and associate member - since September, 2014

Associate Professor at Université d'Auvergne - May, 2013 - September, 2014

Post Doc - January, 2013 - April, 2013

 

Associate Professor at Université d'Auvergne - Since February, 2015

Post Doc - January, 2015 - February, 2015

 

Alumni:

  • Ajad Chhatkuli – PhD Student (September 2013 – August 2016) - Research topic: The Multiple-View Geometry of Deformable Surfaces

 

  • Dr Rahat Khan – Post Doc (November 2013 - October 2015) - Research topic: 3D Reconstruction for Deformable Surfaces from Mutiple Visual Cues with Application to Laparoscopy

 

 

  • Julia Allirot – Project Manager (January 2013 – September 2014)

 

  • Shrinivasan Sankar - Intern (February-July, 2016 - Researc topic: Does detection and recognition help seeing deformable objects in 3D?)

 

  • Ronan Bureau - Intern (March-August, 2016 - Researc topic: Creating digital and physical models to validate deformable 3D reconstruction and aid endoscopic training)

 

  • Héla Hadhri   - Intern   (February - July, 2015 - Researc topic: Realtime Shape-from-Template Using Inverse Composition and the Isometric Local Analytic Solution)

 

  • Tom François - Intern (February - July, 2015 - Researc topic: Fast Methods to Prune Candidates for Shape-from-Template with an Object Database)

 

  • Kristina Prokopetc – Intern (February – June 2014 – Master in Computer Vision, Université de Bourgogne – Research topic: Building a Fallopian Tube Detector for Computer Assisted Intervention)

 

  • Jules Gorny – Intern (March – August, 2014 - Master 2 Technology for Medicine, Université d'Auvergne - Research topic: A Software Framework to Anonymize and Store Medical Image Data)

 

  • Adèle Limosino - Intern (January - April, 2014 - Master 2 ETIRES, Université Sorbonne Paris 1 - Mission report: New medical imaging techniques: seeing better or seeing further?)

 

  • Ludivine Fortin - Intern (January - April, 2014 - Master 2 ETIRES, Université Sorbonne Paris 1 - Mission report: New medical imaging techniques: seeing better or seeing further?)

 

  • Mathilde Lancelot - Intern (January - April, 2014 - Master 2 ETIRES, Université Sorbonne Paris 1 - Mission report: New medical imaging techniques: seeing better or seeing further?)

 

  • Pierre Bruyer – Intern (April – August, 2013 - Master 2 Technology for Medicine, Université d'Auvergne - Master thesis title: Motion based segmentation and 3D reconstruction of the uterus)

 

 

Lectures and talks:

• Keynote, European Workshop on Deformable Object Manipulation  Lyon, Mar.’14

Title: Shape-from-Template

http://www.innorobo.com/en/

 

• Half-day workshop on computer vision, Institut de Recherche en Informatique de Toulouse (IRIT), 2014

Title: Shape-from-Template

 

VTR@MICCAI – Tutorial on Visual tracking and 3D reconstruction for computer-assisted interventions – Nagoya, Sep. ’13

Title: Shape-from-Template in Gynecologic Surgery

http://www.miccai2013.org/

 

Keynote, BMVA Technical Meeting « Reconstructing a Dynamic World »  London, Jul. ’13

Title: Shape-from-Template: Single-View Template-Based 3D Reconstruction

http://bmva_reconstruction.eventbrite.co.uk/

 

Joint GdR ISIS/IG Meeting AC3D (Acquiring and Compressing 3D Objects) – Porquerolles Island, May ’13

Title: Concept and Implementation of Augmented Reality in Gynecologic Laparoscopy

http://www.i3s.unice.fr/~fpayan/gdr-isis.php

 

SurVis – First International Workshop on Surgical Vision – Karlsruhe, May ’13

Title: Shape-from-Template in Gynecologic Surgery

http://ranger.uta.edu/~gianluca/survis/

 

Université de Limoges, Maths-Computer Science Colloquium, Jun.’13

Title: 3D Computer Vision by Shape-from-Motion

http://www.xlim.fr/dmi

 

Institut Camille Jordan, Lyon, Apr. ’13

Title: Projective Geometry in 3D Computer Vision

http://math.univ-lyon1.fr/

 

Augmented Reality for Deformable Surfaces with Application to Laparoscopy

2:00 Master and Doctoral lecture at LabEx CIMI’s first image processing summer school, Saint-Lary, Jun. ’13

http://www.cimi.univ-toulouse.fr/

 

Publications

  

Particle-SfT: a Provably-Convergent, Fast Shape-from-Template Algorithm
E. Özgür and A. Bartoli
International Journal of Computer Vision, accepted October 2016.
 
 
A System for Augmented Reality Guided Laparoscopic Tumor Resection with Quantitative Ex-vivo User Evaluation
T. Collins, P. Chauvet, C. Debize, D. Pizarro, A. Bartoli, N. Bourdel, M. Canis
CARE'16 - Proceedings of the Third International Workshop on Computer Assisted and Robotics Endoscopy at MICCAI, Athens, Greece, October 2016.
 
Can we Jointly Register and Reconstruct Creased Surfaces by Shape-from-Template Accurately?
M. Gallardo, T. Collins and A. Bartoli
ECCV'16 - Proceedings of the European Conference on Computer Vision, Amsterdam, the Netherlands, October 2016.
 
Dense, Robust and Real-time 3D Tracking of Deformable Organs in Monocular Laparoscopy
T. Collins, A. Bartoli, N. Bourdel and M. Canis
MICCAI'16 - Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention, Athens, Greece, October 2016.
 
Using Shading and a 3D Template to Reconstruct Complex Surface Deformations
M. Gallardo, T. Collins and A. Bartoli
BMVC'16 - Proceedings of the British Machine Vision Conference, York, UK, September 2016.
 
Isometric Non-Rigid Shape-from-Motion in Linear Time
S. Parashar, D. Pizarro and A. Bartoli
CVPR'16 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, June 2016.
 
Inextensible Non-Rigid Shape-from-Motion by Second Order Cone Programming
A. Chhatkuli, D. Pizarro, T. Collins and A. Bartoli
CVPR'16 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, June 2016.
 
A Stable Analytical Framework for Isometric Shape-from-Template by Surface Integration
A. Chhatkuli, D. Pizarro, A. Bartoli and T. Collins
IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted April 2016.
 
Stratified Structure-from-Motion for Planar Scenes and Affine Cameras
T. Collins and A. Bartoli
IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted March 2016.
 
Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives
D. Pizarro, R. Khan and A. Bartoli
International Journal of Computer Vision, 119(2):93-109, September 2016.
 
As-Rigid-As-Possible Volumetric Shape-from-Template
S. Parashar, D. Pizarro, A. Bartoli and T. Collins
ICCV'15 - Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, December 2015.
 
T. Collins, A. Bartoli, N. Bourdel and M. Canis
MICCAI'15 - Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention, Munich, Germany, October 2015.
 
Realtime Shape-from-Template: System and Applications
T. Collins and A. Bartoli
ISMAR'15 - Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, Fukuoka, Japan, September 2015.
 
M. Gallardo, D. Pizarro, A. Bartoli and T. Collins
CVPR'15 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, June 2015.
 
A. Malti, A. Bartoli and R. Hartley
CVPR'15 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, June 2015.
 
K. Prokopetc, T. Collins and A. Bartoli
IPMI'15 - Proceedings of the biennial international conference on Information Processing in Medical Imaging, Isle of Skye, Scotland, UK, June 2015.
 
A. Bartoli, T. Collins and D. Pizarro
Computer Vision and Image Understanding, 135:141-156, June 2015.
 
A. Bartoli, Y. Gérard, F. Chadebecq, T. Collins and D. Pizarro
IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(10):2099-2118, October 2015.
 
T. Collins and A. Bartoli
International Journal of Computer Vision, 109(3):252-286, September 2014.
 
A. Chhatkuli, D. Pizarro and A. Bartoli
BMVC'14 - Proceedings of the British Machine VIsion Conference, Nottingham, UK, September 2014.
 
R. Khan, D. Pizarro and A. Bartoli
ECCV'14 - Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, September 2014.
 
T. Collins and A. Bartoli
ECCV'14 - Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, September 2014.
 
T. Collins, P. Mesejo and A. Bartoli
ECCV'14 - Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, September 2014.
 
F. Brunet, A. Bartoli and R. Hartley
Computer Vision and Image Understanding, 125:138-154, August 2014.
 
A. Chhatkuli, D. Pizarro and A. Bartoli
CVPR’14 – Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, Ohio, USA, June 2014. 
 
A. Bartoli, D. Pizarro and T. Collins
ICCV'13 - Proceedings of the International Conference on Computer Vision, Sydney, Australia, December 2013.
 

J. Braux-Zin, R. Dupont and A. Bartoli
ICCV'13 - Proceedings of the International Conference on Computer Vision, Sydney, Australia, December 2013.

Combining Features and Intensity for Wide-Baseline Non-Rigid Surface Registration
J. Braux-Zin, R. Dupont and A. Bartoli
BMVC'13 - Proceedings of the British Machine Vision Conference, Bristol, UK, September 2013.  

Isowarp and Conwarp: Warps that Exactly Comply with Weak-Perspective Projection of Deforming Objects for 3D Reconstruction
D. Pizarro, A. Bartoli and T. Collins
BMVC'13 - Proceedings of the British Machine Vision Conference, Bristol, UK, September 2013.

Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity
A. Malti, R. Hartley, A. Bartoli and J.-H. Kim
CVPR'13 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, Oregon, USA, June 2013.

Template-Based Isometric Deformable 3D Reconstruction with Sampling-Based Focal Length Self-Calibration
A. Bartoli and T. Collins
CVPR'13 - Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, Oregon, USA, June 2013.

Publications