FastPoints2Mesh

Data-Driven Inference of Mesh-based Representations for Deformable Objects from Unstructured Point Clouds

Started
June 1, 2023
Status
In Progress
Share this project

Abstract

Combining robotic manipulation with computer vision holds great promise for automating daily tasks of ever-increasing complexity. However, versatile manipulation of complex or deformable objects is still beyond current robotic capabilities, primarily because fast and robust shape estimation for soft materials remains a challenging and largely open problem. We propose to tackle this challenge with a physics-informed mesh representation, building on recent advances in machine learning to achieve real-time point cloud to mesh reconstruction. Recent methods make it efficient and effective to learn from point cloud representation and show robustness to input perturbation and corruption. We aim to reconstruct in real-time the mesh of a deforming object by bringing suitable physical properties such as deformability, inertia, and density conservation into the loss function of the point cloud to mesh reconstruction neural network. We will procure a large-scale simulated dataset and enhance it with real-world data for training neural networks. The simulated dataset will be based on datasets of CAD object models (ShapeNet, PartNet, etc.) that are available online. The resulting pre-trained network is expected to greatly improve both the surface reconstruction accuracy and the frame rate of real-time mesh reconstruction. Such a quick and accurate meshing solution will be an important foundation for computer vision and robotics. The tracking of complex objects in unstructured environments is crucial for dexterous robotic manipulation and real-world interactions, which will strongly impact many application domains, including medicine, logistics, food processing, and agriculture.

People

Collaborators

SDSC Team:
Firat Ozdemir
Alessandro Maissen
Luis Salamanca
Mathieu Salzmann

PI | Partners:

ETH Zurich, Soft Robotics Lab:

  • Prof. Robert Katzschmann
  • Hehui Zheng
  • Chenyu Yang

More info

ETH Zurich, Computational Robotics Lab:

  • Prof. Stelian Coros
  • Miguel Angel Zamora Mora

More info

description

Motivation

Recent advancements in computer vision could allow for robotic manipulation of complex objects (see Fig. 1 for some robotic manipulators). Along this line, inferring functional and physical properties of previously unseen objects from visual data play a key role to assess effective strategies for manipulation.

Proposed Approach / Solution

The project explores fast point cloud to mesh reconstruction of both soft robot arms (manipulators) and objects of interest (objects in the scene). A parallel effort is for generating a large-scale dataset with complex objects being manipulated. This dataset includes two types of data. One is the real-world collected data with both daily objects such as plush toys shown in Figure 2, and 3D printed deformable objects for easy reproduction [1]. The other is the simulated data based on large-scale 3D model datasets (ShapeNet, PartNet, etc.) that are publicly available online. This would allow for optimizing data-driven models that can infer functional and/or physical properties based on vision alone. We are also exploring approaches for simulating capture systems in an effort to bridge the gap between simulated and real-world data.

Impact

Fast mesh reconstruction is an important foundation for tracking and interacting with complex objects in unstructured scenes. Its impact would surpass the robotics field alone, with applications in medicine, logistics, food processing and agriculture. The generated dataset can promote future novel work from the community and allow for benchmarking findings.

Figure 1: A glimpse into robotic manipulators developed by the collaborators at SRL and CRL of ETH Zurich.
Figure 2: Real-world collected plush toy data. The deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction of the object. The deformable objects are poked by a Husky dual-arm robot.

Gallery

Annexe

Additional resources

Bibliography

  1. Obrist, Jan, et al. "PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics." arXiv preprint arXiv:2410.07688 (2025).

Publications

Related Pages

More projects

MAGNIFY

In Progress
Machine learning Assisted larGe scale quaNtIfication of building energy FlexibilitY
Energy, Climate & Environment

SPI-GreenFjord

In Progress
Energy, Climate & Environment

SPI-PAMIR

In Progress
Energy, Climate & Environment

TREMA

Completed
Transforming real estate management with AI
Engineering

News

Latest news

First National Calls: 50 selected projects to start in 2025
March 12, 2025

First National Calls: 50 selected projects to start in 2025

First National Calls: 50 selected projects to start in 2025

50 proposals were selected through the review processes of the SDSC's first National Calls.
AIXD | Generative AI toolbox for architects and engineers
January 22, 2025

AIXD | Generative AI toolbox for architects and engineers

AIXD | Generative AI toolbox for architects and engineers

Introducing AIXD (AI-eXtended Design), a toolbox for forward and inverse modeling for exhaustive design exploration.
Smartair | An active learning algorithm for real-time acquisition and regression of flow field data
May 1, 2024

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

We’ve developed a smart solution for wind tunnel testing that learns as it works, providing accurate results faster. It provides an accurate mean flow field and turbulence field reconstruction while shortening the sampling time.

Contact us

Let’s talk Data Science

Do you need our services or expertise?
Contact us for your next Data Science project!