MFASS-v1: What Three Splice-Variant Rankers Put in the First 100

In a queue of 100 variants, how many will each method rank that disrupt exon recognition? MFASS, the multiplexed functional assay of splicing using Sort-seq, measures exon inclusion in lab-built minigene reporters.

On this holdout, a baseline trained on 19,409 MFASS variants put 62 experimentally positive variants in its first 100. SpliceAI put 64 in its first 100 scored variants, and Pangolin put 65; neither specialist was fitted to this run's MFASS training arm.

The observed counts are close. None of the three paired precision-at-100 comparisons has a 95% interval that excludes zero.

Across the full ranked lists, the result looks less close. In this run, Pangolin reached average precision (AP) 0.389 and AUROC 0.876 on the variants it scored, compared with 0.299 and 0.806 for SpliceAI, and 0.286 and 0.768 for the baseline. In paired group resampling, Pangolin's AP differences from both other methods and all three AUROC differences have intervals above zero. A method can rank the long list better while the evidence remains inconclusive at a particular queue size.

The label here is an experimental one. In Chong and colleagues' MFASS study, a positive variant disrupted exon recognition in the reporter.

The assay did not measure patient RNA or clinical outcomes. The 100-place queue is an analytic capacity scenario, not a measured laboratory workload. No foundation-model candidate was scored in this first run.

Table 1. Point metrics use each method's own scored subset; coverage uses the original 8,324-variant holdout. AP is scikit-learn average precision. Values come from the baseline result, SpliceAI result and Pangolin result at benchmark commit edf5b5c. The benchmark repository and result files are public.

That denominator column is part of the result. Pangolin's 0.650 and SpliceAI's 0.640 are measured on different available lists. Subtracting them would not produce a paired comparison. Before returning to the 100-place question, it helps to see how the assay cohort, biological holdout and score coverage were constructed.

A functional label with a firm boundary

Chong and colleagues used multiplexed functional assay of splicing using Sort-seq, or MFASS, to assay ExAC-derived variants in minigene reporters. Their published cohort contains 27,733 variants and 1,050 large-effect splice-disrupting variants (SDVs).

The classification requires a decrease in exon inclusion relative to the wild-type reporter, with a delta-inclusion index at or below −0.5 and wild-type inclusion of at least 0.5. Without that wild-type condition, an already excluded exon could be called a new disruption too easily. The endpoint is a defined change in reporter exon recognition.

The paper's byline and PubMed author record list Rockie Chong as first author. Its publisher summary and some subsequent reference lists mistakenly say “Cheung”; Chong is the name used here.

The author data repository provides a cleaned SNV table and a separate functional-annotation table. This reanalysis keeps rows with category == "mutant" and a nonmissing strong_lof label. The rebuilt cohort reconciles to the published 27,733 and 1,050 totals.

The paper's 2,198 exons describe the original mutant set; the filtered benchmark cohort has 2,185 eligible annotated exons. Those numbers refer to different stages, so a table that collapses them into one denominator would be wrong. The pinned cohort builder and the two author tables define the version used here.

All 27,733 retained benchmark rows have a 170-base sequence field, from which the baseline takes local k-mers. SpliceAI and Pangolin score each variant against GRCh38 genomic reference sequence and transcript annotation. Their scores use genomic context, while the reference labels come from the physical reporter. This comparison across molecular contexts supports a claim about ranking the MFASS endpoint; endogenous tissue splicing requires separate evidence.

The cohort's roughly 3.8% positive rate also gives the 100-place queue a useful scale. The held-out arm has 315 SDVs among 8,324 variants.

A blind selection of 100 variants from a list at that prevalence would contain around four positives in expectation; all three reported queues have 62 to 65. The unresolved question is the difference between these already informative rankings. The group structure determines how much confidence to put in a two- or three-positive gap.

Figure 1. Original assay-context schematic. An SDV requires variant-minus-wild-type inclusion change at or below −0.5 and wild-type inclusion at or above 0.5 in the reporter. The genomic rankers use GRCh38 and transcript annotation, while patient RNA and clinical outcome were not measured. Sources: Chong et al. and the pinned MFASS-v1 scorer configuration.

The diagram makes the distinction between a genomic predictor's input and a reporter-derived label visible.

The same variant appears in different molecular contexts. MFASS is valuable because its large, experimentally labelled collection lets us ask a ranking question with enough rare positives to inspect. Tissue-dependent expression, transcript choice and downstream clinical consequences require separate measurements.

A biological holdout, not a random row holdout

A random variant-level split can put several variants from the same exon on both sides of a training boundary. A feature model might then recognise local sequence, boundary position or conservation patterns from an exon it has already seen.

Grouping only by exon closes that route, but two exons from the same gene can still straddle train and test. To make the trained baseline's test more demanding, the canonical split-v2 groups connected exon and gene relationships. Every variant connects its exon ID and gene ID in a graph; whole connected components go to one split arm.

Consider variants in exon A and exon B, both assigned to gene G. The two strings A|G and B|G look like independent groups when the columns are concatenated, although both share G.

An exon annotated to more than one gene can bridge still more records, and the graph retains those transitive connections. The cohort has 2,267 apparent concatenated exon-gene keys but 1,590 connected components. That difference changes what it means for a test exon or gene to be unseen by the supervised model.

Figure 2. Original toy graph, not observed MFASS network structure. Shared exon or gene relationships make one connected train-or-test component even when concatenated keys differ. The real cohort has 2,267 naive keys and 1,590 connected groups. Sources: the pinned split manifest and splitter.

The fixed split manifest records grouping keys ensembl_id and ensembl_gene_id and seed 20260914. Training contains 19,409 variants, 735 SDVs and 1,127 connected groups. Test contains 8,324 variants, 315 SDVs and 463 groups.

Test labels selected the whole-group assignment with prevalence closest to the cohort from 200 seeded draws; model predictions did not select it. The intervals below are conditional on this chosen split. All comparisons use the connected split; the older exon-only split is a different, looser holdout.

For uncertainty, those 463 connected groups are more defensible resampling units than 8,324 freely exchangeable variants. The paired bootstrap later resamples whole groups. Missing specialist scores reduce the common intersection to 454 groups for any pair with SpliceAI and 461 for Pangolin versus the baseline. An uncertainty interval that treated every row as independent would tell a different, overconfident story about the top 100.

Connected grouping addresses one identifiable leakage route for the model trained here. It does not purge a specialist's prior training corpus, undo past public evaluation on MFASS, or make every variant within one group independent of its neighbours. The holdout says that the MFASS-trained baseline did not receive labels from test exon-and-gene components in this run.

It also gives every scorer the same primary test cohort. That is a useful and auditable claim, but a narrower one than declaring the assay novel to all three methods.

Three scorers with different advantages

The supervised baseline uses distances to exon boundaries, reference and alternate allele identity, phyloP and phastCons conservation, and 3-mer composition in a 21-base window. A gradient-boosted tree model (HistGradientBoostingClassifier) fits those features to the 19,409 MFASS training variants and scores all 8,324 held-out rows. It has a clear advantage for this endpoint: it learned directly from MFASS labels on the train arm. Its result represents a feature model with assay supervision.

SpliceAI, from Jaganathan and colleagues at Illumina predicts changes to splice sites from long genomic sequence context. This run uses package 1.3.1, the official five-model ensemble and maximum of its four acceptor/donor gain/loss delta scores.

The 50-base distance flag limits the search for changed sites that contribute to the variant score; the model's input context extends to 10 kilobases. The run sets masking off (-M 0, SpliceAI's default), uses a GRCh38 primary-assembly FASTA, and selects the package's bundled grch38 gene annotation, derived from GENCODE v24 canonical transcripts. The official repository documents the score reduction, annotation bundle and flags.

Pangolin, from Tony Zeng and Yang Li at the University of Chicago predicts splice-site usage changes from genomic sequence. Here its official twelve-model ensemble is reduced to the maximum absolute predicted usage change among reported sites, with a 50-base search radius and mask=False.

That mask setting is explicit because Pangolin's documented default is masked. This benchmark uses the same GRCh38 primary-assembly FASTA but a database built from GENCODE v44 with Pangolin's upstream database script. Pangolin's tissue-related outputs are model predictions.

Table 2. Configuration of this run, from the pinned benchmark scorer code and results, the SpliceAI documentation, Pangolin ensemble source and Pangolin database script. This run's record lacks the installed Pangolin package commit; an upstream pin would define a new rerun until its scores are reconciled.

The specialists were used without fitting to this benchmark's MFASS training arm, but it would be misleading to call MFASS a wholly untouched external test. CADD-Splice previously evaluated SpliceAI on MFASS, and the Pangolin paper evaluated both Pangolin and SpliceAI on the assay. These publications do not demonstrate that a particular held-out exon was part of a checkpoint's training labels.

They do establish prior public evaluation on MFASS. This article adds a connected holdout and a common 100-place reporting protocol, not the first exposure of either specialist to MFASS.

The annotation mismatch belongs in the same comparison paragraph. SpliceAI's bundled GRCh38 annotation is v24-derived; Pangolin's is v44. Annotation can change gene eligibility, coverage and score context. The observed gap is therefore between model-plus-annotation configurations, and the annotation contribution has an unknown direction.

Matching the annotation and rerunning on shared scored variants is necessary before assigning the whole Pangolin versus SpliceAI gap to model architecture. The fact that both reported configurations are unmasked removes one obvious flag mismatch but does not remove this one.

The full list separates; the first 100 do not

Precision@100 matches the analytic queue scenario but ignores everything after rank 100. AP summarizes precision as recall changes across the list; the benchmark uses scikit-learn's noninterpolated average_precision_score. The measures can disagree because moving positives in the long tail changes AP and AUROC without necessarily moving them across the rank-100 boundary.

This AP definition is also why I do not copy older MFASS comparison figures into Table 1. Pangolin's published MFASS evaluation used a different score construction and calculated precision-recall area with auc, whereas this run uses maximum variant-level score reductions and scikit-learn average_precision_score. Its previously published MFASS numbers are legitimate results for that published protocol; they are not interchangeable with a grouped holdout and different metric definition here. Reanalysis should say what changed rather than present a new table as a contradiction of the original paper.

Table 1 uses each method's own scored subset, and the specialist subsets have different positive counts. For a paired comparison, the benchmark intersects variants scored by both methods. Table 3 reports the observed candidate-minus-reference difference on that common subset, alongside a percentile 95% interval from 2,000 seeded draws resampling connected groups.

At precision@100, these observed differences equal subtraction of the Table 1 values: two, three and one additional SDVs per 100. The bootstrap resample means differ from those observed effects and are not used as point estimates.

Table 3. Observed paired differences at exactly 100 ranks and grouped percentile 95% intervals from 2,000 draws. Each resample keeps approximately the same top-list fraction (100 divided by the pair's common-list size), so its realised depth was 80 to 132 for pairs with SpliceAI and 80 to 125 for Pangolin versus the baseline. Intervals condition on this split and fitted baseline; they include no baseline retraining variation or adjustment for the nine comparisons. Sources: SpliceAI minus baseline, Pangolin minus baseline and Pangolin minus SpliceAI, benchmark commit edf5b5c.

Figure 3. Dots are observed paired differences on shared scored variants; bars are grouped percentile 95% intervals from 2,000 draws. Every precision@100 comparison and SpliceAI-minus-baseline AP interval spans zero; both Pangolin AP intervals and all AUROC intervals lie above zero. Sources: SpliceAI versus baseline, Pangolin versus baseline and Pangolin versus SpliceAI.

Both Pangolin AP intervals and all three AUROC intervals exclude zero. SpliceAI's AP interval against the baseline spans −0.040 to +0.056, leaving its direction unresolved. Every precision@100 interval crosses zero, including Pangolin minus SpliceAI at −0.039 to +0.076. Those intervals allow differences larger than the one-positive point gap in either direction.

Crossing zero is not an equivalence result. To conclude that the methods perform equivalently at 100, we would need a predeclared acceptable difference and an equivalence analysis precise enough to exclude differences outside that margin. Neither is present for this three-method first run.

It is accurate to report the observed 62, 64 and 65 queues; it is not accurate to certify a reliable queue winner. At a different capacity the result might change, which is why a future comparison should choose capacity and an improvement margin before opening its candidate scores.

Where the rankings become less tidy

Distance from an exon boundary is an obvious place to ask whether aggregate rankings hide local behaviour. The benchmark stratifies the 8,194 variants that all three methods scored, including 308 positives. In the band more than 30 bases from the nearest boundary, 1,914 variants contain only 31 SDVs.

Its observed AUROC is 0.786 for the baseline, 0.744 for SpliceAI and 0.844 for Pangolin. This distal band was noticed after inspecting four bands, and no grouped subgroup-comparison interval was computed. Distance is also one of the baseline's fitted features, so the changed ordering is confounded with its assay-trained supervision and cannot establish a distance interaction.

Table 4. Descriptive AUROC in distance bands on the three-method common subset, from the pinned subgroup result.

Figure 4. Descriptive AUROC on the 8,194-variant, 308-SDV three-method shared subset, stratified by nearest exon-boundary distance. This plot has no uncertainty bars or interaction test. Source: pinned subgroup result.

A future assay with more distal SDVs and matched annotations could test whether the ordering persists.

Coverage is another source of a changed task. SpliceAI scored 8,194 of 8,324 test variants; its 130 unscored rows were logged as having no overlapping annotated gene. Pangolin scored 8,301 and skipped 23 under a catch-all reason covering no gene, reference mismatch or unsupported input. Its artifact does not disaggregate those causes. The baseline scored all 8,324.

SpliceAI's scored set includes 308 of the holdout's 315 SDVs; Pangolin's includes 314. The unscored-row artifacts matter for deployment planning alongside the score table. A workflow needs a fallback or an explicit unranked bucket for a variant the scorer declines to score.

The paired analysis compares ranking where both methods scored a variant. A method might have good AP among eligible variants and still leave important cases out of its queue. Deployment should report ranking and coverage together.

Elapsed time is useful but must be read as workflow time. In the recorded macOS arm64 run, baseline feature construction, fit and prediction totalled 0.523 seconds. SpliceAI model/reference loading plus test scoring totalled 4,463.805 seconds; Pangolin loading and test scoring totalled 13,664.821 seconds.

The baseline's recorded per-input denominator is the full 27,733-row cohort because its workflow trains and predicts; each specialist's denominator is the 8,324 test inputs. These numbers do not support a pure neural-inference speed ratio or a portable cost prediction. They do say that the reported specialist workflows required substantially more wall time on this machine, and a production choice should be timed with its actual batching, hardware and reference assets.

A tested core path and a specialist provenance gap

The public benchmark snapshot at commit edf5b5c0b83bec27975e8c8a30be88e7a3581a52 was inspected on 15 September 2026. The two pinned author-data tables are snv_data_clean.txt and snv_func_annot.txt at commit 9a8e4f27106be52aeb11acad27f95f5cded663a8; their SHA-256 values are a637ca0e307e66ff48811ec7efa22b9ce453bc7883b04f0cacb867f7283132d8 and 71a857fe647c4e68acbb41ca61e959c47e1176de89b1442bd6ca1772aa60d5a1. Both files are necessary because the second carries gene and conservation information used by the split and baseline. The core path was executed on the current benchmark code, with fresh outputs redirected to an ignored validation directory.

With uv and Python 3.11 or later installed, these commands start from a fresh public checkout and were tested in this writing pass. Compare the two printed digests with the values above before proceeding. The two --key flags are essential: omitting them creates the older exon-only split.

git clone https://github.com/timini/rewire-benchmarks.git
cd rewire-benchmarks
git checkout edf5b5c0b83bec27975e8c8a30be88e7a3581a52
mkdir -p benchmarks/mfass/data/repro
curl -fL --retry 3 -o benchmarks/mfass/data/snv_data_clean.txt \
  https://raw.githubusercontent.com/KosuriLab/MFASS/9a8e4f27106be52aeb11acad27f95f5cded663a8/processed_data/snv/snv_data_clean.txt
curl -fL --retry 3 -o benchmarks/mfass/data/snv_func_annot.txt \
  https://raw.githubusercontent.com/KosuriLab/MFASS/9a8e4f27106be52aeb11acad27f95f5cded663a8/processed_data/snv/snv_func_annot.txt
shasum -a 256 benchmarks/mfass/data/snv_data_clean.txt benchmarks/mfass/data/snv_func_annot.txt
uv sync --package mfass
uv run --no-sync mfass-build \
  --raw benchmarks/mfass/data/snv_data_clean.txt \
  --annot benchmarks/mfass/data/snv_func_annot.txt \
  --out benchmarks/mfass/data/repro/cohort.tsv
uv run --no-sync mfass-split \
  --cohort benchmarks/mfass/data/repro/cohort.tsv \
  --key ensembl_id --key ensembl_gene_id \
  --test-frac 0.3 \
  --seed 20260914 \
  --out benchmarks/mfass/data/repro/split-v2.tsv
uv run --no-sync mfass-baseline \
  --cohort benchmarks/mfass/data/repro/cohort.tsv \
  --split benchmarks/mfass/data/repro/split-v2.tsv \
  --capacity 100 --seed 20260914 \
  --out benchmarks/mfass/data/repro/baseline.json

The expected reconciliation is 27,733 variants and 1,050 SDVs in the cohort, 1,590 connected groups, and 8,324 test variants with 315 SDVs in 463 groups. The baseline should report rounded precision@100 0.620, AP 0.286 and AUROC 0.768. A changed digest or count means this is a different input or split and should be investigated before comparing scores.

The rebuilt cohort matched the author's existing local cohort byte for byte; that comparison file is not committed. Readers can instead verify the two public source hashes and printed cohort counts.

The connected split matched the committed split byte for byte, and the baseline's rounded point metrics matched. Three paired mfass-compare commands were executed against the committed baseline and specialist prediction TSVs, with 2,000 whole-group draws and seed 20260914; their output JSONs matched the committed pairwise JSONs byte for byte. This verifies data preparation, the supervised baseline and comparison calculation from archived scores. Specialist score generation was not independently rerun in this article pass.

The paired calculation can be repeated from those archived full-run prediction tables. For a fresh full specialist run, replace each specialist path with its new full-run .predictions.tsv and regenerate the reported table from the new JSON.

uv run --no-sync mfass-compare \
  --baseline benchmarks/mfass/results/baseline-kmer-position.predictions.tsv \
  --candidate benchmarks/mfass/results/spliceai-1.3.1.predictions.tsv \
  --capacity 100 --draws 2000 --seed 20260914 \
  --out benchmarks/mfass/data/repro/compare-baseline-vs-spliceai.json
uv run --no-sync mfass-compare \
  --baseline benchmarks/mfass/results/baseline-kmer-position.predictions.tsv \
  --candidate benchmarks/mfass/results/pangolin-maskFalse.predictions.tsv \
  --capacity 100 --draws 2000 --seed 20260914 \
  --out benchmarks/mfass/data/repro/compare-baseline-kmer-position-vs-pangolin-maskFalse.json
uv run --no-sync mfass-compare \
  --baseline benchmarks/mfass/results/spliceai-1.3.1.predictions.tsv \
  --candidate benchmarks/mfass/results/pangolin-maskFalse.predictions.tsv \
  --capacity 100 --draws 2000 --seed 20260914 \
  --out benchmarks/mfass/data/repro/compare-spliceai-1.3.1-vs-pangolin-maskFalse.json

The distance-band table can also be regenerated from those committed predictions. This command reproduced the committed subgroups-canonical.json byte for byte:

uv run --no-sync mfass-subgroups \
  --cohort benchmarks/mfass/data/repro/cohort.tsv \
  --methods \
    baseline=benchmarks/mfass/results/baseline-kmer-position.predictions.tsv \
    spliceai=benchmarks/mfass/results/spliceai-1.3.1.predictions.tsv \
    pangolin=benchmarks/mfass/results/pangolin-maskFalse.predictions.tsv \
  --capacity 100 \
  --out benchmarks/mfass/data/repro/subgroups-canonical.json

Source downloads and rerun outputs stay under the ignored benchmarks/mfass/data/ tree, leaving the committed split and result JSON intact. Exact source hashes matter because a revision to a variant table, gene annotation or conservation field could change connected components and the fitted baseline. The full scorer path adds large reference files and software environments; the public checkout includes the archived prediction and result files needed to replay these comparisons.

For full specialist scoring, the pinned benchmark scorer implementations require a GRCh38 primary-assembly FASTA, GENCODE v44 GTF and Pangolin database, model dependencies and checkpoints. SpliceAI uses its bundled v24-derived annotation for this reported run; the v44 database belongs to Pangolin and a future matched-annotation test.

The following full-run commands were inspected against current CLI flags but were not executed in this article pass. They assume the reference FASTA and database have been prepared using the linked GENCODE files and Pangolin database script. A rerun with the shown current Pangolin package pin would be a new run, because the installed commit for the historical result was not recorded.

uv sync --package mfass --extra spliceai --extra pangolin
uv pip install --python .venv/bin/python \
  'git+https://github.com/tkzeng/Pangolin.git@5cf94b8db938c658391b4305cd7ce33297d44ff7'
uv run --no-sync mfass-spliceai \
  --cohort benchmarks/mfass/data/repro/cohort.tsv \
  --split benchmarks/mfass/data/repro/split-v2.tsv \
  --ref benchmarks/mfass/data/ref/GRCh38.primary_assembly.genome.fa \
  --annotation grch38 --distance 50 --mask 0 \
  --out benchmarks/mfass/data/repro/spliceai-full.json
uv run --no-sync mfass-pangolin \
  --cohort benchmarks/mfass/data/repro/cohort.tsv \
  --split benchmarks/mfass/data/repro/split-v2.tsv \
  --ref benchmarks/mfass/data/ref/GRCh38.primary_assembly.genome.fa \
  --db benchmarks/mfass/data/ref/gencode.v44.annotation.db \
  --distance 50 --mask False \
  --out benchmarks/mfass/data/repro/pangolin-full.json

The existing Pangolin result lacks the installed upstream package commit. Checkpoint identity and scores would need reconciliation before treating a new full scorer run as a reproduction of the existing one. The tested path above replays committed prediction TSVs; independently rerunning specialist score generation remains open work.

What the first run lets us decide

For the MFASS minigene endpoint and these configurations, Pangolin's whole-list ranking separates from the MFASS-trained baseline and SpliceAI on AP and AUROC in grouped paired comparisons. The specialists were not fitted to this run's 19,409 MFASS training labels. SpliceAI's AUROC interval against the trained baseline lies above zero in this fixed-split, unadjusted analysis, although its lower bound is only +0.0023; its AP interval crosses zero.

At the original 100-place operating point, every paired interval crosses zero. The observed queues of 62, 64 and 65 positives are useful planning numbers, yet this holdout has not established which method reliably fills that queue best.

The next controlled run should keep the connected holdout, match specialist annotations, record Pangolin package and checkpoint provenance, regenerate coverage and pairwise tables on shared scored variants, and expand the distance analysis with more positive examples. If a foundation-model candidate is later added, its queue capacity and acceptable improvement margin can be declared before its scores are opened. This run did not record a predeclared improvement margin; adding one now cannot make the existing comparison prospective. No foundation model has yet been measured by MFASS-v1, so the specialist-versus-foundation-model question remains open.

Sources and artifacts

• Chong et al., MFASS functional assay and author processed-data repository.
• Jaganathan et al., SpliceAI and official software documentation.
• Zeng and Li, Pangolin and official software documentation.
• Rentzsch et al., CADD-Splice and prior MFASS evaluation, GENCODE human release 44 and scikit-learn average precision definition.
• MFASS-v1 benchmark code, split and result snapshot, commit edf5b5c0b83bec27975e8c8a30be88e7a3581a52.