[generate] Run custom generation code from the Hub

[gante]

What does this PR do?

👉 This is a draft PR for feature discussion 👈

This PR enables the definition of custom decoding methods on the hub, sharing the same interface as generate. In a nutshell, if trust_remote_code=True and recipe="user/repo_name" are passed to generate, the arguments to generate are forwarded to a custom generate function, defined in generate.py in that repo.

Why does this matter

With the popularity of LLMs, LLM-derived models (like VLMs), and test-time compute techniques, there is an explosion of what we can do regarding decoding methods. Instead of having a complex process to include a certain technique in transformers, this would enable developers and researchers to build their own decoding methods and share them with the world with zero additional Python dependencies.

The Hub-hosted decoding techniques would also get all benefits from the Hub, including proper credit to the creator (downloads/likes), a place to discuss the method, ...

Working Example

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, recipe="joaogante/test_generate_from_hub", trust_remote_code=True)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

(generate.py in joaogante/test_generate_from_hub)

---

TODO

Before this PR is ready to be merged, the following are missing:

• A documentation page
• Tests
• A few actual examples on the hub

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.