Whisper fine-tuning resources I keep coming back to

When I first decided to fine-tune Whisper, I did what anyone does: googled around, opened forty browser tabs, read through a dozen tutorials of varying quality, and eventually realized I kept returning to the same four resources. Everything else was either outdated, incomplete, or assumed you already understood the HuggingFace training ecosystem well enough that you didn't need a tutorial. I gathered those four go-to resources into a small GitHub repo, and this post is basically a guided tour through why these specific resources earned their place.

Why fine-tuning tutorials matter more than you'd think

Whisper fine-tuning isn't like fine-tuning a text LLM where you throw some JSONL at a LoRA adapter and call it a day. Audio models have their own set of gotchas: sample rate mismatches, feature extraction pipelines, the distinction between CTC and sequence-to-sequence training, dataset format requirements, and the ever-present question of how much training data is "enough." Good tutorials that walk through these pitfalls are genuinely valuable, and bad ones can waste hours of your time and GPU credits.

Resource 1: The Hugging Face blog post

The Hugging Face blog post on fine-tuning Whisper is the single best starting point. It's written by Sanchit Gandhi, who works on the Transformers audio pipeline, so it's authoritative in a way that random Medium articles aren't. It covers the complete process: dataset preparation with proper column naming, feature extraction, tokenization, defining the data collator, setting up the Seq2SeqTrainer, and evaluation with WER metrics. What I particularly appreciate is that it explains why each step is necessary, not just how to do it. You come away understanding the architecture, not just blindly copying code.

Resource 2: The Google Colab notebook

The Colab notebook is the interactive companion to that blog post. It's the same author, same approach, but in a format where you can execute each cell and inspect the intermediate outputs. I found this invaluable when I was debugging dataset formatting issues --- being able to see exactly what the feature extractor produces, what the tokenized labels look like, and how the data collator shapes batches made the whole pipeline much more tangible. Colab's free GPU tier is enough for experimenting with tiny and base models, though you'll want to upgrade for anything larger.

Resource 3: The Paperspace Gradient notebook

The Paperspace Gradient IPU notebook is a bit of an oddball inclusion, but I keep it around because it demonstrates fine-tuning on Graphcore's IPU hardware rather than NVIDIA GPUs. This is interesting from an architectural perspective --- IPUs handle the training computation differently, with a focus on data parallelism that can be faster for certain model sizes. Even if you never use IPU hardware, reading through how the same training process adapts to different accelerators gives you a deeper understanding of what's actually happening under the hood.

Resource 4: Modal's ASR fine-tuning example

The Modal ASR example is what directly inspired my own fine-tuning script. Modal takes a fundamentally different approach from the notebook-based tutorials: instead of running cells interactively, you define your training environment and pipeline as code, deploy it, and trigger runs from the command line. The environment is fully reproducible, the GPU spins up only when you need it, and costs are dramatically lower than maintaining a dedicated instance. My Modal-Whisper-Finetune-Script repo is essentially an evolution of this example, expanded to support all five Whisper variants with independent caching and configurable hyperparameters.

Why a repo for four links?

I know what you're thinking --- a GitHub repo for four links seems like overkill. And honestly, yeah, it kind of is. But I've found that the simplest repos are often the most useful ones. Every time I start a new fine-tuning experiment or want to reference how the data collator works, I open this repo instead of searching through browser history or HuggingFace docs. It's a personal bookmark collection that happens to be public. Sometimes the most valuable thing you can share isn't original code but a curated pointer to the resources that actually helped you.

The broader landscape

Whisper fine-tuning has gotten easier since I started, but the ecosystem is still fragmented. There are tutorials scattered across HuggingFace blogs, Medium posts, YouTube videos, and random GitHub repos. Some are for the original Whisper, some for Faster-Whisper, some for distilled variants. The four resources I've gathered here are the ones that consistently work, are well-maintained, and cover the standard HuggingFace Transformers approach that most people will want. If you're just getting started with Whisper fine-tuning, start with the blog post, play with the Colab notebook, and then graduate to Modal when you're ready to do serious training runs.

Check it out here: Whisper Fine-Tuning Resources on GitHub.