gugarosa commited on
Commit
c44dab7
·
verified ·
1 Parent(s): 932b33c

Restore <|endoftext|> (100257) as a stop token in generation_config

Browse files

## Summary
Set `generation_config.json` `eos_token_id` back to `[100257, 100265]`, restoring
`<|endoftext|>` (100257) alongside `<|im_end|>` (100265) as a stop token. This reverts
only the `eos_token_id` change from #21 while keeping that PR's other improvements
(chat template fix, `pad_token` -> `<|dummy_85|>`).

## Problem
With `eos_token_id` set to `100265` only, the model can fail to stop: it produces a
correct answer and then continues emitting unrelated text until `max_tokens` is reached
(reported as "random tokens until the max token limit").

## Root cause
phi-4 ends many (typically terse) assistant turns with `<|endoftext|>` (100257) rather
than `<|im_end|>` (100265). Since #21, `generation_config.json` only lists `100265`, so
when the model emits `100257` no stop criterion fires; decoding runs past the turn
boundary (the following `<|im_start|>user<|im_sep|>` gets stripped on output, leaving a
stray `user`) and free-runs until the token cap.

## Evidence
- Reproduced with bare HF `model.generate()` at temperature 0 (greedy). Identical on
transformers 4.47 and 5.13, so it is not a transformers-version issue.
- Reproduced with vLLM 0.18.0 and 0.24.0 — byte-identical output to HF.
- Next-token distribution right after "The capital of France is Paris.":
`P(<|endoftext|>=100257) = 0.72` vs `P(<|im_end|>=100265) = 0.27`, so greedy decoding
deterministically selects the token the current config does not stop on.
- With `eos_token_id = [100257, 100265]`, generation stops cleanly on all three engines.

## Scope
Only `generation_config.json` is changed. `config.json` and the tokenizer files are left
as-is; this is the minimal change that both `transformers` and vLLM honor for stopping.

Files changed (1) hide show
  1. generation_config.json +4 -1
generation_config.json CHANGED
@@ -1,7 +1,10 @@
1
  {
2
  "_from_model_config": true,
3
  "bos_token_id": 100257,
4
- "eos_token_id": 100265,
 
 
 
5
  "pad_token_id": 100349,
6
  "transformers_version": "4.47.0"
7
  }
 
1
  {
2
  "_from_model_config": true,
3
  "bos_token_id": 100257,
4
+ "eos_token_id": [
5
+ 100257,
6
+ 100265
7
+ ],
8
  "pad_token_id": 100349,
9
  "transformers_version": "4.47.0"
10
  }