Credibility-Aware Multi-Modal Fusion Using Probabilistic Circuits

Overview

Decision-making in fields like healthcare depends on synthesizing diverse data sources — MRIs, electronic health records, lab results. Multimodal data offer rich representations and multiple views of an underlying phenomenon, but these sources frequently contain noise, gaps, and contradictions. This work targets late fusion scenarios, where individual per-modality models each produce a prediction, and those predictions are then combined into a single answer.

We introduce a principled, probabilistic way to combine predictions from sources that aren't equally reliable — weighting each modality by how credible it is for the input at hand, rather than trusting every source equally.

Key contributions

A principled probabilistic framework for combining information from multiple data sources with varying reliability levels.
A theoretically grounded credibility measure linked to conditional entropy in unimodal predictions, enabling dependable late fusion.
Two algorithmic variants — Direct-PC and Credibility Weighted Mean — suited to different domain requirements.
Experimental validation showing the framework correctly models cross-modal interactions and accurately estimates credibility.

Defining credibility through consistency

Rather than relying on hand-tuned confidence scores, we define a source as reliable if it provides credible information — measured by how much a modality's presence changes the model's predictive distribution. Concretely, credibility is the KL divergence between the prediction made with a modality present and the prediction made without it:

Credibility(X) = D_KL( P(Y | X) || P(Y | X \ X) )

A high-credibility modality shifts the prediction substantially when included; a low-credibility one barely moves it. This ties credibility directly to conditional entropy, giving it a clean theoretical grounding rather than treating it as a heuristic.

Computing credibility with probabilistic circuits

Evaluating that formula requires conditioning on arbitrary subsets of modalities being present or absent — exactly the kind of partial-observability query that's expensive for most joint models. We use probabilistic circuits (PCs) — a model class that represents a joint distribution as a computational graph of simple distributions — because they support exact conditionals under partial observability efficiently and differentiably.

Each modality is first passed through its own predictor to produce a unimodal predictive distribution. A probabilistic circuit then models the joint distribution over these per-modality outputs and the target variable, which is what makes it possible to query "what would the prediction be without modality X?" in closed form.

Two fusion methods

Direct-PC (DPC) reads the fused prediction directly off the circuit's joint distribution over unimodal outputs and the target — no separate weighting step.

Credibility Weighted Mean (CWM) instead combines the individual unimodal predictions in a weighted average, using each modality's credibility score as its weight.

Both are fully differentiable, so the whole pipeline — per-modality predictors and the fusion circuit — can be trained end-to-end with backpropagation.

Experiments

We validated the approach on AV-MNIST, a dataset pairing image and audio modalities that both predict the same spoken/displayed digit. We injected controlled noise into individual modalities and tracked how the model's credibility assignments responded.

The result: the noisy modality's credibility score dropped in proportion to the injected noise, while the clean modality's credibility rose to compensate — confirming that the framework is sensitive to actual information quality, not just which modality happens to be present.

Citation

@inproceedings{sidheekh2024credibility,
    title = {Credibility-aware Multi-Modal Fusion Using Probabilistic Circuits},
    author = {Sidheekh, Sahil and Tenali, Pranuthi and Mathur, Saurabh and Blasch, Erik and Kersting, Kristian and Natarajan, Sriraam},
    booktitle = {International Conference on Artificial Intelligence and Statistics},
    year = {2025}
}

Supported by AFOSR (FA9550-23-1-0239), ARO (W911NF2010224), and the DARPA Assured Neuro Symbolic Learning and Reasoning program (HR001122S0039).

← Back to blog