Extracting activity from large 4D whole-brain image datasets

July 1, 2021
In Progress
Share this project


Whole-brain recordings hold promise to revolutionize neuroscience. In the last decade, innovations in fast 3D microscopy, protein engineering, genetics, and microfluidics have allowed brain researchers to read out calcium activity at high temporal resolution from a large number of neurons in the brains of Caenorhabditis elegans, Danionella translucida, Hydra, and zebrafish simultaneously. This technology is considered to be a game changer for neuroscience as it leaves far fewer variables hidden than when only a small fraction of neuronal activities could be recorded. Many fundamental and challenging questions of neuroscience can now be pursued:

  • What global brain activity determines an organism's responses to stimuli?
  • How are decisions computed by networks of neurons?
  • What is the idle activity of an unstimulated brain?

The field suffers from a critical bottleneck. Neuronal activities are recorded as local intensity changes in 4D microscopy images. Extracting this information for a moving animal is very labor-intensive and requires expertise. The promise of whole-brain recordings cannot be fully realized unless the image analysis problem is solved.

There are several challenges:

  1. 3D images are generally difficult to annotate manually.
  2. The worm moves, rotates, bends, and compresses fast.
  3. To avoid blurring, the exposure time and the image quality are limited.
  4. The resolution in the z-direction is low.



SDSC Team:
Benjamin Béjar Haro
Corinne Jones
Guillaume Obozinski
Isinsu Katircioglu

PI | Partners:

Laboratory of the Physics of Biological Systems:

  • Prof. Sahand Rahi
  • Dr. Elif Gençtürk
  • Alice Gross
  • Mahsa Barzegarkeshteli
  • Matthieu Schmidt

More info



The goals of the collaboration consist of identifying specific neurons across 4D images (segmentation & tracking), mapping every pixel in 4D images onto a 3D reference (registration) and speeding up the former tasks for real-time feedback to the animal.


The SDSC will help to design robust, efficient algorithm for tracking a specific set of neurons in videos of freely moving worms, and will propose machine learning techniques to align the worm images within each video and extract the activities of the neurons.


Efficient image analysis techniques would reduce the burden of manual annotation and unleash the growth of the field. Faster image analysis would mean that:– more and a more diverse range of experiments can be performed,– more animals can be analyzed ‘per paper’, making results more statistically rigorous,– more scientists could perform such experiments,– “high-throughput neuroscience” with freely moving animals will become possible,– new questions will become accessible, for example, individual differences between animals cannot be studied in a statistically rigorous way with the few worms that are usually analyzed ‘per paper’.


Figure 1: We aim to identify pieces of neurons or whole neurons in 3D images and track them in time. This can be done by mapping 3D images from different time points onto the same reference 3D image.


Additionnal resources


  1. S. Chaudhary, S. A. Lee, Y. Li, D. S. Patel, and H. Lu. Graphical-model framework for automated annotation of cell identities in dense cellular images. In eLife 10:e60321, Feb. 2021.
  2. S. Kato, H. S. Kaplan, T. Schrödel, S. Skora, T. H. Lindsay, E. Yemini, S. Lockery, and M. Zimmer. Global brain dynamics embed the motor command sequence of Caenorhabditis elegans. In Cell, 163 (3):656–669, Oct. 2015.
  3. J. P. Nguyen, A. N. Linder, G. S. Plummer, J. W. Shaevitz, and A. M. Leifer. Automatically tracking neurons in a moving and deforming brain. In PLOS Computational Biology, 13(5):1–19, 05 2017.
  4. X. Yu, M. S. Creamer, F. Randi, A. K. Sharma, S. W. Linderman, and A. M. Leifer. Fast deep neural correspondence for tracking and identifying neurons in C. elegans using semi-synthetic training. In eLife, 10:e66410, Jul. 2021.


Related Pages

More projects


In Progress
Machine Learning for the Future Circular Collider Design
Big Science Data


In Progress
Real-time cleansing of snow and weather data for operational avalanche forecasting
Energy, Climate & Environment


AI-augmented architectural design
Energy, Climate & Environment


In Progress
Deep statistical learning-based image analysis for measurement of socioeconomic development in sub-Saharan Africa using high-resolution satellite images, and geo-referenced household survey data
Energy, Climate & Environment


Latest news

PassGPT | Using language models to enhance password security
February 6, 2024

PassGPT | Using language models to enhance password security

PassGPT | Using language models to enhance password security

PassGPT is a Large Language Model for password generation trained on leaked passwords, which can outperform existing methods based on generative adversarial networks by guessing twice as many unseen passwords.
ADORE | A benchmark dataset in ecotoxicology to foster the adoption of machine learning
January 24, 2024

ADORE | A benchmark dataset in ecotoxicology to foster the adoption of machine learning

ADORE | A benchmark dataset in ecotoxicology to foster the adoption of machine learning

Applying machine learning to ecotoxicology could help reduce the number of animal tests, costs, and animals sacrificed while preserving the accuracy of the in vivo tests.
License Flowers | Art and AI at SDSC
February 21, 2024

License Flowers | Art and AI at SDSC

License Flowers | Art and AI at SDSC

An adventure to create art using AI to raise awareness on code licenses

Contact us

Let’s talk Data Science

Do you need our services or expertise?
Contact us for your next Data Science project!