cc @Rocketknight1

Hi @chuyishang, this should be fixed now! Please try updating to the latest version, or installing from main if that doesn't work.

@Rocketknight1
There is a new issue introduced from this fix.

When using multi-round few-shot such as

[
    {"role": "user", "content": f"hi 0"},
    {"role": "assistant", "content": f"bye: 0"},
    {"role": "user", "content": f"hi 1"},
    {"role": "assistant", "content": f"bye: "}
]

The rendered chat with tokenizer.apply_chat_template(messages, tokenize=False, continue_final_message=True, add_generation_prompt=False) will actually completely remove the last round if the tokenizer strips the text. Giving you

<bos><start_of_turn>user
hi 0<end_of_turn>
<start_of_turn>model
bye: 

This is because the last round "bye: " is only found in the penultimate round since the last round is now just "bye:" (no white space).

I'm not sure what would be the best way to remedy this. Maybe max(rindex(final_message), rindex(final_message.strip()))?

@TK-21st yes, you're completely correct and I hadn't considered that! Working on a high-priority fix now

Hi @TK-21st, I've opened a PR to fix this at #36404. After investigation, the error you've found is very template-specific. Can you tell me what template you were using? I'll add a test using it to the transformers codebase so we don't regress on this

Hi @TK-21st, I've opened a PR to fix this at #36404. After investigation, the error you've found is very template-specific. Can you tell me what template you were using? I'll add a test using it to the transformers codebase so we don't regress on this

Great thanks!
I was using google/gemma-2-2b-it