MFASS splice-variant prioritisation
A corrected baseline and one zero-cost local DNABERT-2 protocol on MFASS. The held-out ranking result is complete.
MFASS tests exon recognition in a minigene assay. Of 27,733 variants, 8,324 were held out in 463 exon/gene groups; 315 were splice-disrupting. The review capacity is 100 variants. Read the assay paper →
Held-out results
| Method | P@100 | AP | AUROC | Coverage |
|---|---|---|---|---|
| Corrected k-mer / position baseline | 0.610 | 0.286 | 0.778 | 8324/8324 |
| DNABERT-2 117M · frozen pair embeddings | 0.030 | 0.045 | 0.550 | 8324/8324 |
| SpliceAI 1.3.1 | 0.640 | 0.299 | 0.806 | 8194/8324 |
| Pangolin · mask=False | 0.650 | 0.389 | 0.876 | 8301/8324 |
P@100 is the fraction of confirmed disruptions among the first 100 variants; AP is average precision across the ranking; AUROC is the area under the receiver operating characteristic curve.
The corrected baseline found 61 confirmed disruptions in its first 100. This frozen DNABERT-2 pair-embedding and fixed head found 3; both scored every held-out variant. The unchanged SpliceAI and Pangolin specialist runs use genomic context and have incomplete coverage, so their point metrics are on their scored subsets.
Paired difference
Each candidate minus the corrected baseline, on variants both scored. Intervals are 95% group-bootstrap intervals from 2,000 draws over whole exon/gene groups.
| Candidate | Common variants | P@100 [95% interval] | AP [95% interval] | AUROC [95% interval] |
|---|---|---|---|---|
| SpliceAI minus baseline | 8,194 | 0.030 [-0.068, 0.107] | 0.009 [-0.039, 0.061] | 0.028 [-0.004, 0.064] |
| Pangolin minus baseline | 8,301 | 0.040 [-0.041, 0.120] | 0.102 [0.063, 0.141] | 0.098 [0.073, 0.125] |
| DNABERT-2 minus baseline | 8,324 | -0.580 [-0.713, -0.464] | -0.241 [-0.304, -0.182] | -0.228 [-0.300, -0.160] |
After the correction, SpliceAI's paired AUROC interval crosses zero; the earlier v1 claim of clear separation is withdrawn. Pangolin still separates from the baseline on AP and AUROC, while P@100 is uncertain. The tested DNABERT-2 protocol falls below the baseline on all three measures.
Protocol and limits
Assay-oriented 21 bp k-mer window, exon position, allele identity and conservation features; gradient-boosted trees trained on the MFASS training split.
Masked mean of frozen last hidden states for 170 bp reference and mutant sequences; concatenate reference and mutant-minus-reference embeddings; fixed balanced L2 logistic head trained only on the MFASS training split. The model checkpoint is pinned to revision b5ae377faa37. The full run took 10 minutes using M4 CPU, 16 GB RAM; no paid compute was used.
The baseline and DNABERT-2 head are supervised on 19,409 MFASS training variants, while the specialists are zero-shot on the assay. MFASS measures a functional reporter assay rather than patient RNA or clinical pathogenicity. Exact sequence and exon overlap with DNABERT-2 pretraining has not been checked. SpliceAI and Pangolin use different annotation releases, so their gap cannot be attributed to models alone. This one fixed embedding/head protocol does not establish DNABERT-2's best achievable result.