Data-Driven Inference of Mesh-based Representations for Deformable Objects from Unstructured Point Clouds

June 1, 2023
In Progress
Share this project


Combining robotic manipulation with computer vision holds great promise for automating daily tasks of ever-increasing complexity. However, versatile manipulation of complex or deformable objects is still beyond current robotic capabilities, primarily because fast and robust shape estimation for soft materials remains a challenging and largely open problem. We propose to tackle this challenge with a physics-informed mesh representation, building on recent advances in machine learning to achieve real-time point cloud to mesh reconstruction. Recent methods make it efficient and effective to learn from point cloud representation and show robustness to input perturbation and corruption. We aim to reconstruct in real-time the mesh of a deforming object by bringing suitable physical properties including deformability, inertia, and density conservation into the loss function of the point cloud to mesh reconstruction neural network. This approach ensures that the objects’ physical properties always lie in a physically realistic range at all times. We expand on a large-scale simulated dataset enhanced with real-world data to train our neural network. The simulated dataset will be based on datasets of CAD object models (ShapeNet, PartNet, etc.) that are available online. The resulting pre-trained network is expected to greatly improve both the surface reconstruction accuracy and the frame rate of real-time mesh reconstruction. Such a quick and accurate meshing solution will be an important foundation for computer vision and robotics. The tracking of complex objects in unstructured environments is crucial for dexterous robotic manipulation and real-world interactions, which will strongly impact many application domains, including medicine, logistics, food processing, and agriculture.



SDSC Team:
Firat Ozdemir
Luis Salamanca

PI | Partners:

ETH Zurich, Soft Robotics Lab:

  • Prof. Robert Katzschmann
  • Hehui Zheng
  • Chenyu Yang

More info

ETH Zurich, Computational Robotics Lab:

  • Prof. Stelian Coros
  • Miguel Angel Zamora Mora

More info



Recent advancements in computer vision could allow for robotic manipulation of complex objects (see Fig. 1 for some robotic manipulators). Along this line, inferring functional and physical properties of previously unseen objects from visual data play a key role to assess effective strategies for manipulation.

Proposed Approach / Solution

The project will explore fast point cloud to mesh reconstruction of both soft robot arms (manipulators) and objects of interest (objects in the scene). A parallel effort is going to be generating a large-scale dataset with complex objects being manipulated. This dataset will include two types of data. One is the real-world collected data with both daily objects such as plush toys shown in Fig. 2, and 3D printed deformable objects for easy reproduction. The other is the simulated data based on large-scale 3D model datasets (ShapeNet, PartNet, etc.) that are publicly available online. This would allow for optimizing data-driven models that can infer functional and/or physical properties based on vision alone.


Fast mesh reconstruction is an important foundation for tracking and interacting with complex objects in unstructured scenes. Its impact would surpass the robotics field alone, with applications in medicine, logistics, food processing and agriculture.

Figure 1: A glimpse into robotic manipulators developed by the collaborators at SRL and CRL of ETH Zurich.
Figure 2: Real-world collected plush toy data. The deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction of the object. The deformable objects are poked by a Husky dual-arm robot.



Additional resources



Related Pages

More projects


In Progress
Interpretable and Robust Machine Learning for Mobility Analysis
No items found.


In Progress
Feature Learning for Bayesian Inference
No items found.


In Progress
Personalized epidural electrical stimulation of the lumbar spinal cord for clinically applicable therapy to restore mobility after paralyzing spinal cord injury
No items found.


In Progress
Directed Imitation During Vocal Learning
No items found.


Latest news

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data
May 1, 2024

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

We’ve developed a smart solution for wind tunnel testing that learns as it works, providing accurate results faster. It provides an accurate mean flow field and turbulence field reconstruction while shortening the sampling time.
The Promise of AI in Pharmaceutical Manufacturing
April 22, 2024

The Promise of AI in Pharmaceutical Manufacturing

The Promise of AI in Pharmaceutical Manufacturing

Innovation in pharmaceutical manufacturing raises key questions: How will AI change our operations? What does this mean for the skills of our workforce? How will it reshape our collaborative efforts? And crucially, how can we fully leverage these changes?
Efficient and scalable graph generation through iterative local expansion
March 20, 2024

Efficient and scalable graph generation through iterative local expansion

Efficient and scalable graph generation through iterative local expansion

Have you ever considered the complexity of generating large-scale, intricate graphs akin to those that represent the vast relational structures of our world? Our research introduces a pioneering approach to graph generation that tackles the scalability and complexity of creating such expansive, real-world graphs.

Contact us

Let’s talk Data Science

Do you need our services or expertise?
Contact us for your next Data Science project!