build error from tokenizers dependency

[joprice]

I'm seeing the same rust build error reported here huggingface/tokenizers#1485, which was fixed in huggingface/tokenizers#1355 (tokenizers-0.14.1). Running pip install --dry-run transformers==4.34.0 shows that the version is the one including the fix. Is it possible to update transformers to a more recent version?

Otherwise, there is this workaround suggested: RUSTUP_TOOLCHAIN=1.72.0 pip install tokenizers==0.13.2, but it another error

  error: package `clap v4.5.26` cannot be built because it requires rustc 1.74 or newer, while the currently active rustc version is 1.72.0```.

[joprice]

I was able to build by setting RUSTFLAGS="-A invalid_reference_casting" to ignore the warning, but seems like since it can lead to UB, it should be addressed eventually.