Voice-Health Claim Audit — transparency dashboard

The result, in four steps

Each step removes something the model was getting for free, and asks what survives. Every number here is read from the artifact named beside it.

1. F1/F3 — patient age sex duration rms reaches 0.8747 recording-level AUC; WavLM (ens_rank3) reaches 0.7443. The audio model loses to one demographic variable. n=8 repeats × 5 folds on 1,679 speakers · bench_svd_wavlm_mean_std.json
2. F4 — projecting out the speaker-identity subspace costs WavLM more AUC than removing more variance any other way: 24–39% of its above-chance headroom is who is speaking. n=10 · V2_speaker_subspace.json, shares computed over the ranks whose CI excludes zero
3. F7a — with age matched to a 0.77-year gap, age-only collapses 0.8747 → 0.5534. The confound is genuinely removed, not assumed away. n=10 · V1_ssl_vs_handcrafted.json
4. F7b — on that matched subset eGeMAPS 0.6496 beats WavLM 0.6227 (paired difference -0.0269, 95% CI [-0.032, -0.022], 10/10 seeds). 88 handcrafted features beat 1536 learned ones.

What survives once demographics and identity are stripped is AUC ≈ 0.65 — real, well above chance, far below the ~0.85 the literature reports here, and nowhere near clinical usefulness.

Each step links to the run that produced it: bench-wavlm · v2-speaker-subspace · v1-ssl-vs-handcrafted.

The three tiers of this site

This page is the master. Every claim on it drills down twice, and each tier is generated from the same declared artifacts, so the three cannot disagree.

• Master (here) — the four-step result, the run ledger, the findings ledger, the corpus and scoreboard tables, the audits, and the source provenance.
• Per-hypothesis registry — 7 pre-registered hypotheses, each with its claim, falsifier and predicted effect rendered above any result. 3 of 7 have never been executed and are carried as visible debt.
• Per-run ledger — 8 artifact-backed runs, one page each, rendering the artifact in full: every interval, the power arithmetic, and the reproduction provenance. These pages also state what they lack — there is no 7-step reasoning entry and no experiment_log.jsonl in this repository.
• The dataset landscape — the 25 corpora the field uses, scored on whether a result on each could mean anything.

The idea

A person speaks; a model listens and predicts a health condition. The appeal is that the sensor is a microphone everyone already owns. Two mechanisms make it plausible rather than magical. First, the larynx is the instrument — a vocal-fold paralysis, a polyp, swelling, a tumour or surgical scarring changes the sound directly. Second, speech is a motor act needing breath control, timing and fine neuromuscular coordination, so neurological and respiratory disease can leave traces even when the larynx is healthy. The first mechanism is short and well-evidenced; the second is a long causal chain and correspondingly weaker. The literature also claims depression, diabetes and heart failure from voice — this program tests none of those, for the reason in the next block.

The corpus

The primary corpus is the Saarbrücken Voice Database, a clinical archive from Saarland University Hospital. Each participant records 14 short vocalisations: the sustained vowels /a/, /i/, /u/ at normal, high, low and rising-falling pitch, plus one spoken German sentence.

How many diseases can it detect? Not 70.

Sorted by how many speakers carry each diagnosis — not how many recordings, since a condition with 200 clips from 8 people is an 8-person condition:

19 of the 70 conditions are represented by a single speaker. So the honest answer is:

• One task is properly powered — binary healthy vs pathological (1,008 vs 671 speakers). Every headline number on this page refers to that task and no other.
• At most 12 named conditions clear a ≥30-speaker floor, and would need one-vs-rest treatment with wide intervals. The largest are Rekurrensparese (139), Hyperfunktionelle Dysphonie (126), Laryngitis (79), Psychogene Dysphonie (70).
• 58 conditions cannot be modelled at all at any defensible sample size.

And note what those conditions are: almost all are dysphonias and structural larynx disorders — the category where the sound changes because the sound-producing organ changed. This corpus contains essentially no systemic disease. A model trained here detects disordered voice, not disease in general, and describing it otherwise is a category error.

Why this program exists

Published SVD results report UAR in the mid-80s. But healthy participants here average 27 years and patients 49, so patient age alone reaches ROC-AUC 0.87 without hearing a single audio sample (F1 below). Any classifier that quietly learns "older ⇒ patient" inherits that score for free. The interesting quantity is therefore not the accuracy but the margin above the demographic baseline on the identical folds — a number the field does not currently report. Measuring it is the entire purpose of this repository.

Every count in this section is recomputed at build time from data/interim/svd/manifest.csv; none is hand-entered.

The rest of the field → the dataset landscape

SVD is one corpus of many. The companion page catalogues every dataset the field uses to claim a disease can be heard in a voice — 25 corpora across 7 disease families (larynx, Parkinson's, dementia, dysarthria, respiratory, mental health, multi-phenotype), 8 of them released or audited in 2026 — each scored on the question the published surveys do not ask: could a result on this dataset mean anything? Spoiler: 5 of the 25 can never carry a generalisation claim at all.

• Every number carries its n and its tier. SCREENING means n ≤ 3 and may never be called a result; EVALUATION means n ≥ 8 with the full statistical contract. A cell with no measurement says not yet measured — it is never filled with a guess.
• A confound baseline outranks a model. Age, sex, duration and loudness are fitted first; an audio model may claim only the margin above the strongest of them. On this page the audio model currently loses that comparison.
• Speaker-disjoint or it does not count. Splits group on speaker id. A corpus with no real speaker id (COUGHVID) cannot carry a generalisation claim at all, and is marked in red below.
• Negative and blocked entries are the product. The ledger's value is that it reports the claims that did not survive and the corpora that cannot support a claim, in the same tables and with the same detail as everything else.

Two inputs disagree about the same corpus

data/PREPROCESSING_STATUS.md (hand-maintained, last written 2026-07-25 22:39 UTC) states SVD at 667 decoded files from 49 speakers. data/interim/svd/summary.json (machine-written, 2026-07-28 03:12 UTC) states 61,170 decoded from 1,679 speakers.

Every count on this page comes from the JSON. The markdown is stale and is flagged rather than narrated — an earlier build of this page rendered the markdown in one panel and the JSON in the next, so two adjacent elements contradicted each other with nothing marking either. This panel is generated by a comparison, so it disappears on its own once the markdown is refreshed.

What is blocked right now

• SVD is decoded but not complete. 61,170 of 61,178 found files decoded, covering 1,679 speakers against the 1,853 the distributed metadata lists — so 174 speakers have metadata but no audio on this host, and no audio result covers them. (data/interim/svd/summary.json vs F1_demographic_baseline.json)
• Coswara audio is partially decoded. 610 of 684 found files, 72 of 2,746 participants — the host has ~15 GB free disk and the decoder stops cleanly rather than filling the volume. (data/interim/coswara/summary.json)
• COUGHVID can never support an evaluation claim. 34,434 rows, 34,434 unique uuids, zero participant identifiers. Pretraining and out-of-distribution probing only.
• PROCESS-2 is behind a manually-reviewed HuggingFace gate (CognoSpeak/PROCESS-2); Bridge2AI-Voice v3.1.0 needs PhysioNet credentialing plus a separate DACO application for raw audio. Both are paperwork-bound, not compute-bound. (data/ACQUISITION_STATUS.md)

On SVD, patient age alone reaches ROC-AUC 0.8709 without hearing any audio

SCREENING metadata-only · artifact data/raw/svd_meta/voice_data.csv (md5 2ee9852a19ed…) · measured values in autoresearch_results/F1_demographic_baseline.json

Using only the demographic metadata distributed with the Saarbrücken Voice Database — no audio, no features, no model of speech — over n = 2,225 sessions (1,356 pathological / 869 healthy) from 1,853 unique speakers:

Underlying cause — a recruitment asymmetry

Healthy speakers are young volunteers; pathological speakers are older clinic patients. Mean age 28.3 years (healthy) vs 51.0 years (pathological) — a gap of 22.7 years.

A live leakage trap in the same artifact

200 of 1,853 speakers contribute more than one session (max 24 sessions for a single speaker). Splitting SVD at the recording level — the default in any naive train_test_split — leaks those speakers across folds. All work in this repository splits on SprecherID via GroupKFold and asserts speaker-disjointness before every fit.

What this does NOT claim — read before citing

1. This does not show any published result is wrong. UAR and ROC-AUC are different metrics and are not directly comparable; the comparison above is indicative, not a like-for-like contest. A proper audit must recompute the published pipeline's metric on matched splits — that is the next experiment, not this one.
2. Several published pipelines may already control for age, use age-matched subsets, or report demographic baselines. The audit question is whether the margin above the demographic bar is reported at all — not whether these researchers were careless.
3. Metadata-only baselines say nothing about whether voice does carry pathology signal. It very likely does. The claim is strictly about attribution.

Quoted verbatim from FINDINGS.md §F1. The published SVD benchmark is UAR 85.22 (arXiv:2410.10537) — different metric, indicative only.

Reproduce

# metadata (167 KB), then the baseline -- CPU, seconds, no audio required
python scripts/fetch_svd.py --metadata-only
python scripts/audit_demographic_baseline.py --dataset svd

eGeMAPS on the decoded SVD pilot slice does not clear its own age bar

EVALUATION SCOPE-LIMITED — 49 of 1,679 speakers CONFOUND BAR NOT CLEARED · artifact autoresearch_results/bench_svd_egemaps.json · config hash 3ffad10ed6634618 · run commit a640ceb19f59 · 2026-07-25T23:55:48Z · 58 min CPU

Command: run_benchmark.py --corpus svd --backbone egemaps --head all --folds 5 --repeats 10 — opensmile-eGeMAPSv02, 88 features, 5-fold speaker-disjoint × 10 repeated partitions, n=667 recordings / 49 speakers, 20 pathological / 29 healthy speakers.

Sortable: click a column header. CI = 2,000-resample speaker-level bootstrap. All rows: n=667 recordings / 49 speakers.

Result: the strongest confound baseline is age_only at speaker-level AUC 0.7086. The best audio head (gbt) reaches 0.7983, a delta of +0.0897 (95% CI [-0.076, 0.264]). All 5 heads are NOT CLEARED. On this slice, eGeMAPS carries no demonstrated signal above patient age.

What this does NOT claim, and what has since superseded it. This is 49 speakers — a speaker-disjoint test fold holds about ten people and the 95% CI on any accuracy is roughly ±15 pp, visible in the intervals above. It satisfies the repeat contract (10 repeats) and is still a narrow slice; both facts are stated because either alone would mislead. It is not evidence that eGeMAPS fails on full SVD. The same corpus has since been measured at 1,679 speakers with a different representation — see run-bench-wavlm, which reaches the same verdict against the same bar. This section is kept rather than deleted because a superseded null that is quietly removed is indistinguishable from one that was never run (R8).