Correlation-Aware Bayesian Aggregation of Large Language Models with Identifiability and Calibrated, Distribution-Free Uncertainty

International Conference APPLIED STATISTICS 2026, Koper, Slovenia, 21 - 23 September 2026, pp.62, (Summary Text)

Large language models (LLMs) are often asked to interpret the same text, such

as its meaning, tone, or intent. Five such models may agree unanimously on

only one case in five, so merging them reliably is the real problem. Sharing

training data and alignment methods, they tend to err together and to be

overconfident. Classical aggregators such as Dawid-Skene and MACE assume

independent errors, an assumption that fails for correlated LLMs. We develop

a hierarchical Bayesian framework that combines several correlated LLMs

into one calibrated decision with finite-sample, distribution-free error control.

The framework (i) learns how reliable each model is for each class from a

small labelled set, (ii) discounts agreement between models that share failure

modes, so five lookalike votes do not count as five independent ones, (iii) adds

a context-sensitive pragmatic layer whose strength is provably identifiable

from the data rather than hand-tuned, with a credible interval excluding zero,

(iv) calibrates its probabilities and uses conformal prediction to control the

error rate on automatically classified cases, and (v) treats model disagreement

as signal, not noise. We test it on speaker intent in television dialogue. About

10,000 utterances were annotated by five LLMs, 5,000 labelled independently

by two people (Cohen’s 𝜅 = 0.50 on intent). It automatically classifies 61%

of cases at a certified error below 10%, halving the human effort, whereas no

single model certifies any. It also outperforms the best single model (accuracy

0.728 to 0.787, 𝑝 < 0.001) and, while fully interpretable, matches or surpasses a

non-interpretable benchmark. Finally, the framework produces a calibrated

prediction of how often two independent people would disagree on the label

(calibration error 0.085), and extends these gains to figurative meaning. These

results support treating LLM ensembles as correlated multi-agent systems

rather than independent annotators.