Problem whith **tokenizers-0.13.4rc3** compiling Windows 10 Environment 22H2

[amadoui]

I got a problem whith tokenizers-0.13.4rc3 compiling
I am trying on compile following the guide with miniconda on Windows 10 Environment 22H2 (OS Build 19045.4894)
I have the following version installed in miniconda venv:

• conda 24.7.1
• Python 3.10.14 (tried with Python 3.12 / tried with Python 3.11)
• rustc 1.74.1 (a28077b28 2023-12-04) (tried with rustc 1.76 and failed / tried with rustc 1.72.1 and cannot compile because rustc 1.74 is required)

   Compiling tokenizers v0.13.4 (C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\tokenizers-lib)
error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
   --> tokenizers-lib\src\models\bpe\trainer.rs:541:47
    |
537 |                     let w = &words[*i] as *const _ as *mut _;
    |                             -------------------------------- casting happend here
...
541 |                         let word: &mut Word = &mut (*w);
    |                                               ^^^^^^^^^
    |
    = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
    = note: `#[deny(invalid_reference_casting)]` on by default

error: could not compile `tokenizers` (lib) due to previous error
warning: build failed, waiting for other jobs to finish...
Traceback (most recent call last):
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 252, in build_extension
    cargo_messages = subprocess.check_output(
  File "C:\Users\user\.conda\envs\tortoise\lib\subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\user\.conda\envs\tortoise\lib\subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'rustc', '--lib', '--message-format=json-render-diagnostics', '--manifest-path', 'Cargo.toml', '--release', '--features', 'pyo3/extension-module', '--crate-type', 'cdylib', '--']' returned non-zero exit status 101.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 165, in save_modules
    yield saved
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 207, in setup_context
    yield
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 268, in run_setup
    _execfile(setup_script, ns)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 50, in _execfile
    exec(code, globals, locals)
  File "C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\setup.py", line 13, in <module>
    long_description=long_description,
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 168, in run
    cmd = self.call_command('install_lib', warn_dir=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 154, in call_command
    self.run_command(cmdname)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install_lib.py", line 19, in run
    self.build()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\command\install_lib.py", line 110, in build
    self.run_command('build_ext')
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\setuptools_ext.py", line 171, in run
    build_rust.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\command.py", line 100, in run
    self.run_for_extension(ext)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 138, in run_for_extension
    dylib_paths = self.build_extension(ext, self.target)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 261, in build_extension
    raise CompileError(format_called_process_error(e, include_stdout=False))
distutils.errors.CompileError: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release --features pyo3/extension-module --crate-type cdylib --` failed with code 101

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Windows\System32\tortoise-tts\setup.py", line 6, in <module>
    setuptools.setup(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install.py", line 97, in run
    self.do_egg_install()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install.py", line 158, in do_egg_install
    cmd.run(show_deprecation=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 439, in run
    self.easy_install(spec, not self.no_deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 689, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 739, in install_item
    self.process_distribution(spec, dist, deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 788, in process_distribution
    distros = WorkingSet([]).resolve(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 897, in resolve
    dist = self._resolve_dist(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 933, in _resolve_dist
    dist = best[req.key] = env.best_match(
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 1271, in best_match
    return self.obtain(req, installer)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\pkg_resources\__init__.py", line 1307, in obtain
    return installer(requirement) if installer else None
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 712, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 737, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 934, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 1206, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\easy_install.py", line 1192, in run_setup
    run_setup(setup_script, args)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 258, in run_setup
    with setup_context(setup_dir):
  File "C:\Users\user\.conda\envs\tortoise\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 199, in setup_context
    with save_modules():
  File "C:\Users\user\.conda\envs\tortoise\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 178, in save_modules
    saved_exc.resume()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 152, in resume
    raise exc.with_traceback(self._tb)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 165, in save_modules
    yield saved
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 207, in setup_context
    yield
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 268, in run_setup
    _execfile(setup_script, ns)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\sandbox.py", line 50, in _execfile
    exec(code, globals, locals)
  File "C:\Users\user\AppData\Local\Temp\easy_install-urg_k0xs\tokenizers-0.13.4rc3\setup.py", line 13, in <module>
    long_description=long_description,
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 183, in setup
    return run_commands(dist)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\core.py", line 199, in run_commands
    dist.run_commands()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 168, in run
    cmd = self.call_command('install_lib', warn_dir=False)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\bdist_egg.py", line 154, in call_command
    self.run_command(cmdname)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\command\install_lib.py", line 19, in run
    self.build()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\command\install_lib.py", line 110, in build
    self.run_command('build_ext')
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\dist.py", line 950, in run_command
    super().run_command(command)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools\_distutils\dist.py", line 973, in run_command
    cmd_obj.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\setuptools_ext.py", line 171, in run
    build_rust.run()
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\command.py", line 100, in run
    self.run_for_extension(ext)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 138, in run_for_extension
    dylib_paths = self.build_extension(ext, self.target)
  File "C:\Users\user\.conda\envs\tortoise\lib\site-packages\setuptools_rust\build.py", line 261, in build_extension
    raise CompileError(format_called_process_error(e, include_stdout=False))
setuptools.sandbox.UnpickleableException: CompileError('`cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release --features pyo3/extension-module --crate-type cdylib --` failed with code 101')

(tortoise) C:\Windows\System32\tortoise-tts>

Any ideas about how to compile tokenizers-0.13.4rc3 succefully on Windows 10?

[kleethesama]

This #816 (comment) fixed it for me.

[amadoui]

@kleethesama tried your suggestion but i got the same issue when compiling "tokenizers-0.13.4rc3", i suspect the rustc compiler is not suitable for this task but cannot figure out how to deal with it

Compiling tokenizers v0.13.4 (C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\tokenizers-lib)
           Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib\src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"cached-path\"" --cfg "feature=\"clap\"" --cfg "feature=\"cli\"" --cfg "feature=\"default\"" --cfg "feature=\"dirs\"" --cfg "feature=\"esaxx_fast\"" --cfg "feature=\"http\"" --cfg "feature=\"indicatif\"" --cfg "feature=\"onig\"" --cfg "feature=\"progressbar\"" --cfg "feature=\"reqwest\"" -C metadata=571339cac8db871d -C extra-filename=-571339cac8db871d --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern aho_corasick=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libaho_corasick-6e1ceb2730930a08.rmeta --extern cached_path=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libcached_path-b7c63fc96b7b8a96.rmeta --extern clap=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libclap-84048ff9c95fb4cc.rmeta --extern derive_builder=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libderive_builder-2b0b7bf261866a03.rmeta --extern dirs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libdirs-49d8e5d94e87fa5e.rmeta --extern esaxx_rs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libesaxx_rs-48bffd32ccaeb3f5.rmeta --extern getrandom=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libgetrandom-0d5f94c7c56cebb1.rmeta --extern indicatif=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libindicatif-90913102d1e1fe0f.rmeta --extern itertools=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libitertools-a824d246a521d621.rmeta --extern lazy_static=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblazy_static-90ac5ec4ca579963.rmeta --extern log=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblog-e72785d52d978e26.rmeta --extern macro_rules_attribute=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmacro_rules_attribute-c5ccec30f814a496.rmeta --extern monostate=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmonostate-f82384f188902be6.rmeta --extern onig=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libonig-0a91cd76d2a7fb72.rmeta --extern paste=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\paste-0469fbd59fc2e04c.dll --extern rand=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librand-bbef81b04dff903c.rmeta --extern rayon=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon-0692a0d06ad1db6b.rmeta --extern rayon_cond=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon_cond-7ec684bc4c5c804d.rmeta --extern regex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex-5553ea54d73d96cc.rmeta --extern regex_syntax=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex_syntax-7f4550e60d4205c7.rmeta --extern reqwest=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libreqwest-721a7e7c36e12898.rmeta --extern serde=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde-21a5e9db8ed00d2f.rmeta --extern serde_json=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde_json-8457fa567cc6f19d.rmeta --extern spm_precompiled=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libspm_precompiled-2071f0bb46ce086f.rmeta --extern thiserror=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libthiserror-1ea9a51dd83679bd.rmeta --extern unicode_normalization_alignments=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_normalization_alignments-e02e49ff2fe58290.rmeta --extern unicode_segmentation=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_segmentation-10977e716c0679c6.rmeta --extern unicode_categories=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_categories-a7fb754aa5a4a1a4.rmeta -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.52.6\lib -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\bzip2-sys-d909296b23c64ab5\out\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\zstd-sys-ca9050e4fbf40534\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\esaxx-rs-d8450fba8101abff\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\onig_sys-84a28246dad559a1\out`
           Running `rustc --crate-name ndarray --edition=2018 C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\ndarray-0.13.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=0a691bf027808050 -C extra-filename=-0a691bf027808050 --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern matrixmultiply=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmatrixmultiply-05b30d9aee34739d.rmeta --extern num_complex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_complex-5aefd29d72e65d50.rmeta --extern num_integer=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_integer-da28bd3984fda6f1.rmeta --extern num_traits=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libnum_traits-d946bd1460732915.rmeta --extern rawpointer=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librawpointer-5e716fe4b452057e.rmeta --cap-lints allow`
      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib\src\models\bpe\trainer.rs:541:47
          |
      537 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      541 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      error: could not compile `tokenizers` (lib) due to previous error

      Caused by:
        process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib\src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"cached-path\"" --cfg "feature=\"clap\"" --cfg "feature=\"cli\"" --cfg "feature=\"default\"" --cfg "feature=\"dirs\"" --cfg "feature=\"esaxx_fast\"" --cfg "feature=\"http\"" --cfg "feature=\"indicatif\"" --cfg "feature=\"onig\"" --cfg "feature=\"progressbar\"" --cfg "feature=\"reqwest\"" -C metadata=571339cac8db871d -C extra-filename=-571339cac8db871d --out-dir C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps -L dependency=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps --extern aho_corasick=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libaho_corasick-6e1ceb2730930a08.rmeta --extern cached_path=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libcached_path-b7c63fc96b7b8a96.rmeta --extern clap=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libclap-84048ff9c95fb4cc.rmeta --extern derive_builder=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libderive_builder-2b0b7bf261866a03.rmeta --extern dirs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libdirs-49d8e5d94e87fa5e.rmeta --extern esaxx_rs=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libesaxx_rs-48bffd32ccaeb3f5.rmeta --extern getrandom=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libgetrandom-0d5f94c7c56cebb1.rmeta --extern indicatif=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libindicatif-90913102d1e1fe0f.rmeta --extern itertools=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libitertools-a824d246a521d621.rmeta --extern lazy_static=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblazy_static-90ac5ec4ca579963.rmeta --extern log=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\liblog-e72785d52d978e26.rmeta --extern macro_rules_attribute=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmacro_rules_attribute-c5ccec30f814a496.rmeta --extern monostate=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libmonostate-f82384f188902be6.rmeta --extern onig=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libonig-0a91cd76d2a7fb72.rmeta --extern paste=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\paste-0469fbd59fc2e04c.dll --extern rand=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librand-bbef81b04dff903c.rmeta --extern rayon=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon-0692a0d06ad1db6b.rmeta --extern rayon_cond=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\librayon_cond-7ec684bc4c5c804d.rmeta --extern regex=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex-5553ea54d73d96cc.rmeta --extern regex_syntax=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libregex_syntax-7f4550e60d4205c7.rmeta --extern reqwest=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libreqwest-721a7e7c36e12898.rmeta --extern serde=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde-21a5e9db8ed00d2f.rmeta --extern serde_json=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libserde_json-8457fa567cc6f19d.rmeta --extern spm_precompiled=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libspm_precompiled-2071f0bb46ce086f.rmeta --extern thiserror=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libthiserror-1ea9a51dd83679bd.rmeta --extern unicode_normalization_alignments=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_normalization_alignments-e02e49ff2fe58290.rmeta --extern unicode_segmentation=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_segmentation-10977e716c0679c6.rmeta --extern unicode_categories=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\deps\libunicode_categories-a7fb754aa5a4a1a4.rmeta -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.52.6\lib -L native=C:\Users\user\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\bzip2-sys-d909296b23c64ab5\out\lib -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\zstd-sys-ca9050e4fbf40534\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\esaxx-rs-d8450fba8101abff\out -L "native=C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.41.34120\atlmfc\lib\x64" -L native=C:\Users\user\AppData\Local\Temp\pip-install-xemg5ck0\tokenizers_9ae7aa5a45ab44feb7b7128d84e551dc\target\release\build\onig_sys-84a28246dad559a1\out` (exit code: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Successfully built tortoise-tts
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

[robbiemu]

@kleethesama tried your suggestion but i got the same issue when compiling "tokenizers-0.13.4rc3", i suspect the rustc compiler is not suitable for this task but cannot figure out how to deal with it

remember to pip uninstall tokenizers and transformers :) you may also face an issue currently with numpy - I manually set the requirements to the current values (facing same thing in Mac):

issuecomment on #728

[rose07]

https://tts.byylook.com/ai/text-to-speech