Is adding extra tokens supported?

[namgyaaal]

Hello!

I have fine-tuned whisper small via. transformers and I want to use with whisper.cpp. It has extra tokens added with tokenizer.add_tokens() and model.resize_token_embeddings() that are tags outputted alongside speech. Testing it on transformers shows that it works.

Currently generating the model with python models/convert-h5-to-ggml.py safetensors_export_dir ./whisper ./out_model and testing the ggml bin file on a test audio file produces illegible results.

Is there something I'm missing in this process or are additional tokens currently not supported?

Thanks.

[bogdanr]

whisper.cpp does not support extra tokens at the end of the vocabulary. This is the cause of the illegible output.

Why it fails

whisper.cpp does not read the IDs of the special tokens from the model. It calculates them from n_vocab. See src/whisper.cpp:

bool is_multilingual() const {
    return n_vocab >= 51865;
}

int num_languages() const {
    return n_vocab - 51765 - (is_multilingual() ? 1 : 0);
}

Then it moves all task and timestamp tokens by a delta:

const int dt = vocab.num_languages() - 98;

vocab.token_translate  += dt;
vocab.token_transcribe += dt;
vocab.token_solm       += dt;
vocab.token_prev       += dt;
vocab.token_nosp       += dt;
vocab.token_not        += dt;
vocab.token_beg        += dt;

model.resize_token_embeddings() makes vocab_size larger. models/convert-h5-to-ggml.py writes that value as n_vocab:

fout.write(struct.pack("i", hparams["vocab_size"]))

Thus dt becomes too large. Each special token ID moves by the number of tokens that you added:

Model	n_vocab	n_langs	token_translate	token_beg
small, no change	51865	99	50358	50364
small, 4 extra tokens	51869	103	50362	50368
small, 10 extra tokens	51875	109	50368	50374

Two things then go wrong:

1. The decoder uses the wrong IDs for the task tokens and the timestamp tokens.
2. Your new tokens get IDs that are higher than token_beg. The decoder reads them as timestamp tokens, and not as text.

The result is illegible text.

How to confirm this

Look at the model load log. Find the n_langs line. A correct multilingual model shows n_langs = 99. A large-v3 model shows n_langs = 100. A larger number shows that your vocabulary is too large.

What to do

You have two options.

Option 1 — keep the vocabulary size (recommended).

Do not call resize_token_embeddings(). Select tokens that your model does not need, and give them your tag strings. vocab_size then stays at 51865, and all special token IDs stay correct. No change to whisper.cpp is necessary.

Option 2 — change whisper.cpp.

Give is_multilingual() and num_languages() the correct results for your model, or set the special token IDs directly. Then build again. You must apply this patch again after each update.