FLBI

Feature Learning for Bayesian Inference

Started
September 1, 2022
Status
In Progress
Share this project

Abstract

The goal of this project is to use machine learning to find low-dimensional features in high-dimensional noisy data generated by stochastic models or real systems. In the first case, we are interested in the features imprinted on simulated data by the parameters of the stochastic model. In the latter, the interesting features depend on the particular system. In hydrology, one of the domains considered in this project, they are fingerprints of catchment properties in observed time-series of river-runoff. In astrophysics, another considered domain, the features are the parameters of solar dynamo models that govern the magnetic field strength of the sun. In each case, the problem is to disentangle the effect of high-dimensional disturbances from the effects of relevant characteristics (i.e., parameters, fingerprints, etc.). This problem is reminiscent of the problem of finding collective quantities characterizing states of interacting particle systems in statistical mechanics. Variational autoencoders, have been proven to be capable of learning such quantities in the form of order parameters or collective variables, which could then be used to, e.g., enhance Markov chain Monte Carlo samplers or molecular dynamics simulations.

People

Collaborators

SDSC Team:
Simon Dirmeier
Fernando Perez-Cruz

PI | Partners:

Swiss Federal Institute of Aquatic Science and Technology, Mathematical Methods in Environmental Research:

  • Dr. Carlo Albert
  • Dr. Simone Ulzega
  • Alberto Bassi

More info

University of Lugano, Institute of Computational Science:

  • Prof. Dr. Antonietta Mira

More info

description

Motivation

Computational Bayesian inference of parameters or summary statistics is complicated in high-dimensional parameter- or data-spaces. In that case, conventional methods, such as approximate Bayesian computation or sequential Monte Carlo versions thereof, fail to infer correct posterior distributions for a measurement which necessitates the development of efficient methods for these scenarios. A recent approach utilizes deep learning to approximate posterior distributions with neural networks, however, these approaches still do not scale well to high-dimensional data, such as time series data of the magnetic field of solar cycle.

Proposed Approach / Solution

For this project, we develop novel methods for high-dimensional approximate inference, both in parameter- and data-space, utilizing recent developments in neural density estimation and dimensionality reduction. We a) propose several theoretical approaches to solve high-dimensional approximate inference and b) develop high-quality Python libraries implementing these algorithms for applied researchers to use.
One of our recent methods, which we call surjective sequential neural likelihood estimation, uses dimensionality-reducing normalizing flows to more efficiently estimate probability densities which allows for increased accuracy in inferring posterior distributions. On several experimental benchmarks our method outperforms baseline methods while being more computationally efficient w.r.t. the number of trainable parameters and computational speed (Figures 1 and 2).

Impact

We expect of our methods to be of great value for Bayesian inference, since current methodology is not suitable for high-dimensional time series data, for instance time series observations from solar dynamos or other ODE/PDE-models.

Figure 1: Assessing the performance of SSNL on several experimental benchmarks. We compared our method, SSNL, against several baseline methods (SNL, SNRE-C, SNPE-C) in four benchmark models. The x-axis shows the sample size used for training the model, while the y-axis shows the divergence to the true posterior (the lower the better). SSNL either outperforms the baselines or is on par while requiring less trainable parameters due to the dimensionality reduction.
Figure 2: Assessing the performance of SSNL on a complicated solar dynamo model. Applying our dimensionality-reducing method, SSNL, to a solar dynamo shows its advantage over a dimensionality-preserving method (SNL). With only 1000 data points (called “round 1” above) SSNL achieves to locate the posterior distributions correctly around the true parameter value (black dot), while SNL infers a biased posterior distribution. When using 15,000 data points (called “round 15”), both methods achieve similar results.

Gallery

Annexe

Publications

  • Dirmeier, S., Albert, C., & Perez-Cruz, F. (2023). Simulation-based inference using surjective sequential neural likelihood estimation. arXiv preprint arXiv:2308.01054
    doi:10.48550/arXiv.2308.01054
  • Dirmeier, S. (2024). Surjectors: surjection layers for density estimation with normalizing flows. Journal of Open Source Software, 9(94), 6188.
    doi:10.21105/joss.06188
  • Bassi, A., Höge, M., Mira, A., Fenicia, F., & Albert, C. (2024). Learning Landscape Features from Streamflow with Autoencoders. Hydrology and Earth System Sciences Discussions, 2024, 1-30.
    doi:10.5194/hess-2024-47
  • Di Noia, A., Macocco, I., Glielmo, A., Laio, A., & Mira, A. (2024). Beyond the noise: intrinsic dimension estimation with optimal neighbourhood identification. arXiv preprint arXiv:2405.15132.
    doi:10.48550/arXiv.2405.15132

Additional resources

Bibliography

  1. Albert, Carlo, Hans R. Künsch, and Andreas Scheidegger. "A simulated annealing approach to approximate Bayes computations." Statistics and computing 25 (2015): 1217-1232.
  2. Greenberg, David, Marcel Nonnenmacher, and Jakob Macke. "Automatic posterior transformation for likelihood-free inference." International Conference on Machine Learning. PMLR, 2019.
  3. Papamakarios, George, David Sterratt, and Iain Murray. "Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows." The 22nd international conference on artificial intelligence and statistics. PMLR, 2019.

Publications

Related Pages

More projects

ML-L3DNDT

Completed
Robust and scalable Machine Learning algorithms for Laue 3-Dimensional Neutron Diffraction Tomography
Big Science Data

BioDetect

Completed
Deep Learning for Biodiversity Detection and Classification
Energy, Climate & Environment

IRMA

In Progress
Interpretable and Robust Machine Learning for Mobility Analysis
No items found.

STIMO

In Progress
Personalized epidural electrical stimulation of the lumbar spinal cord for clinically applicable therapy to restore mobility after paralyzing spinal cord injury
No items found.

News

Latest news

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data
May 1, 2024

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

Smartair | An active learning algorithm for real-time acquisition and regression of flow field data

We’ve developed a smart solution for wind tunnel testing that learns as it works, providing accurate results faster. It provides an accurate mean flow field and turbulence field reconstruction while shortening the sampling time.
The Promise of AI in Pharmaceutical Manufacturing
April 22, 2024

The Promise of AI in Pharmaceutical Manufacturing

The Promise of AI in Pharmaceutical Manufacturing

Innovation in pharmaceutical manufacturing raises key questions: How will AI change our operations? What does this mean for the skills of our workforce? How will it reshape our collaborative efforts? And crucially, how can we fully leverage these changes?
Efficient and scalable graph generation through iterative local expansion
March 20, 2024

Efficient and scalable graph generation through iterative local expansion

Efficient and scalable graph generation through iterative local expansion

Have you ever considered the complexity of generating large-scale, intricate graphs akin to those that represent the vast relational structures of our world? Our research introduces a pioneering approach to graph generation that tackles the scalability and complexity of creating such expansive, real-world graphs.

Contact us

Let’s talk Data Science

Do you need our services or expertise?
Contact us for your next Data Science project!