FastPoints2Mesh

Data-Driven Inference of Mesh-based Representations for Deformable Objects from Unstructured Point Clouds

Started
June 1, 2023
Status
In Progress
Share this project

Abstract

Combining robotic manipulation with computer vision holds great promise for automating daily tasks of ever-increasing complexity. However, versatile manipulation of complex or deformable objects is still beyond current robotic capabilities, primarily because fast and robust shape estimation for soft materials remains a challenging and largely open problem. We propose to tackle this challenge with a physics-informed mesh representation, building on recent advances in machine learning to achieve real-time point cloud to mesh reconstruction. Recent methods make it efficient and effective to learn from point cloud representation and show robustness to input perturbation and corruption. We aim to reconstruct in real-time the mesh of a deforming object by bringing suitable physical properties such as deformability, inertia, and density conservation into the loss function of the point cloud to mesh reconstruction neural network. We will procure a large-scale simulated dataset and enhance it with real-world data for training neural networks. The simulated dataset will be based on datasets of CAD object models (ShapeNet, PartNet, etc.) that are available online. The resulting pre-trained network is expected to greatly improve both the surface reconstruction accuracy and the frame rate of real-time mesh reconstruction. Such a quick and accurate meshing solution will be an important foundation for computer vision and robotics. The tracking of complex objects in unstructured environments is crucial for dexterous robotic manipulation and real-world interactions, which will strongly impact many application domains, including medicine, logistics, food processing, and agriculture.

People

Collaborators

SDSC Team:
Firat Ozdemir
Alessandro Maissen
Luis Salamanca
Mathieu Salzmann

PI | Partners:

ETH Zurich, Soft Robotics Lab:

  • Prof. Robert Katzschmann
  • Hehui Zheng
  • Chenyu Yang

More info

ETH Zurich, Computational Robotics Lab:

  • Prof. Stelian Coros
  • Miguel Angel Zamora Mora

More info

description

Motivation

‍Recent advancements in computer vision could allow for robotic manipulation of complex objects (see Fig. 1 for some robotic manipulators). Along this line, inferring functional and physical properties of previously unseen objects from visual data play a key role to assess effective strategies for manipulation.

Proposed Approach / Solution

The project explores fast point cloud to mesh reconstruction of both soft robot arms (manipulators) and objects of interest (objects in the scene). A parallel effort is for generating a large-scale dataset with complex objects being manipulated. This dataset includes two types of data. One is the real-world collected data with both daily objects such as plush toys shown in Figure 2, and 3D printed deformable objects for easy reproduction [1]. The other is the simulated data based on large-scale 3D model datasets (ShapeNet, PartNet, etc.) that are publicly available online. This would allow for optimizing data-driven models that can infer functional and/or physical properties based on vision alone. We are also exploring approaches for simulating capture systems in an effort to bridge the gap between simulated and real-world data.

Impact

Fast mesh reconstruction is an important foundation for tracking and interacting with complex objects in unstructured scenes. Its impact would surpass the robotics field alone, with applications in medicine, logistics, food processing and agriculture. The generated dataset can promote future novel work from the community and allow for benchmarking findings.

Figure 1: A glimpse into robotic manipulators developed by the collaborators at SRL and CRL of ETH Zurich.
Figure 2: Real-world collected plush toy data. The deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction of the object. The deformable objects are poked by a Husky dual-arm robot.

Gallery

Annexe

Additional resources

Bibliography

  1. Obrist, Jan, et al. "PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics." arXiv preprint arXiv:2410.07688 (2025).

Publications

Related Pages

More projects

IICU National Data Stream

In Progress
Personalized, data-driven prediction and assessment of infection-related outcomes in Swiss intensive care units (ICUs)
Academia
Public sector

SMARTT

In Progress
Smart sampling for small-angle x-ray scattering Tensor Tomography
Academia

TGSF-DA

In Progress
Data Analysis of the Turbulence and Gravity roles in Stellar Formation
Academia

MOBIKO

In Progress
Mountain Biodiversity Knowledge Graph
Academia

More projects

No results found.
There are no results with this criteria. Try changing your search.
No result

News

Latest news

Elevating International Geneva's Impact with AI
October 21, 2026

Elevating International Geneva's Impact with AI

Elevating International Geneva's Impact with AI

Two AI projects have been selected for funding through the 2026 Geneva AI Initiative, run by the EPFL AI Center with the SDSC.
12 Zurich SMEs join next AI Innovation Program
October 15, 2026

12 Zurich SMEs join next AI Innovation Program

12 Zurich SMEs join next AI Innovation Program

Strong results encourage further AI exploration to simplify operations and boost the competitiveness of SMEs in the Canton of Zurich.
La science des données au service de la neutralité carbone du canton : ENERBAT ouvre la voie
September 17, 2026

La science des données au service de la neutralité carbone du canton : ENERBAT ouvre la voie

La science des données au service de la neutralité carbone du canton : ENERBAT ouvre la voie

News release : ENERBAT, un outil fondé sur les données soutient la politique de rénovation des bâtiments du Canton de Vaud.

Contact us

Would you like to explore how we can work together?

Our team of data scientists and professionals looks forward to engaging in impactful, results-driven collaborations with you.