Overfitting is when a model memorises specific examples instead of learning general patterns. Sometimes called "rote learning" — the model gets very good at recognising things that look like its training data, but anything that looks even slightly different feels wrong to it and it gets confused.

A good model learns the underlying signal. A deepfake detector should learn what makes a synthetic voice sound synthetic — patterns that show up across many different fake-voice methods, not just the specific ones it studied. If it only recognises fake voices that look exactly like the ones it trained on, it has overfit.

The way you spot overfitting is to test the model on examples it has never seen — and ideally on examples that are quite different from what it trained on. If performance drops gracefully, the model is generalising. If it falls off a cliff, the model has overfit.

We tested the detector on four progressively harder challenges. Each step further from what it trained on tells us how well it generalises.

Two things to notice. First — the model degrades gradually, not catastrophically. It doesn't go from 0.69% to 50% (which would mean random guessing on anything new). That tells us it did learn real patterns, not just memorise specific clips.

Second — there's still a big gap. Going from 0.69% on familiar territory to 26.33% on brand new fake-voice technology is a 38× jump. That's not catastrophic, but it's also not great. The model clearly learned features that matter for the kinds of fake voices it studied — and those features don't fully transfer to fake voices made by methods it has never seen.

What works well: When tested on 13 fake-voice methods it had never seen during training, it achieved a 5.55% error rate — roughly 94 out of 100 predictions correct on completely new fakes. It becomes appropriately uncertain on medium-difficulty attacks (66% confidence on A07). And it handles noisy, real-world audio without false-alarming (93.7% confidence on a noisy real voice). These are signs of a model that learned real anti-spoofing patterns, not just memorised its training data.

What doesn't work: Two real problems. First, the model has a complete blind spot for A10 attacks — it classifies the hardest spoof example as "100% authentic," completely wrong. But there's a specific reason: A10 is a Tacotron 2 + WaveRNN system whose output is so natural that even human listeners cannot distinguish it from real speech. The ASVspoof 2019 paper itself confirms that A10's acoustic features literally overlap with authentic speech in feature space. Since our model relies on acoustic representations (Wav2Vec 2.0 features), it faces the same fundamental limit human ears do — there's no acoustic signal to detect.

Second, on the WaveFake dataset (modern neural vocoders like MelGAN and HiFi-GAN — the same technology used in real-world voice cloning today), the error rate jumps to 26.33%. These vocoders produce different artifacts from what the model trained on. Since our project's goal is detecting AI voice cloning broadly, this is a real coverage gap.

What this means: The model is not classically "overfit" in the sense of having memorised its training data — the 5.55% result on unseen attacks proves that. But it does have limited coverage: it learned to detect certain types of synthesis artifacts (the ones present in ASVspoof) and is blind to others (A10, neural vocoders). For the project's stated goal of detecting AI voice cloning broadly, this is a meaningful gap.