tokenizers.apply_chat_template with continue_final_message=True with </think> token

[isamu-isozaki]

System Info

This is very similar to this issue but since it's slightly different I made a new issue. This is on transformers 4.49.0
I was working with deepseek-ai/DeepSeek-R1-Distill-Qwen-32B and I found that the below gives an exception saying substring not found

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("deepseekr1_qwen_32b")
history = [
           {'role': 'assistant', 'content': "</think>"},
]
continue_final_message = True
tokenizer.apply_chat_template(history, tokenize=False, continue_final_message=continue_final_message)

I didn't look too much into the cause but I did notice this happens whenever the model response ends with the token which does happen occasionally(maybe a hallucination)

Who can help?

@ArthurZucker and @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("deepseekr1_qwen_32b")
history = [
           {'role': 'assistant', 'content': "</think>"},
]
continue_final_message = True
tokenizer.apply_chat_template(history, tokenize=False, continue_final_message=continue_final_message)

this will raise an error. The tokenizer folder was made with a folder named deepseekr1_qwen_32b with the tokenizer json files from https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/tree/main

The output is an exception saying substring not found

Expected behavior

No exceptions and no output