Windows system cannot install pip install - r requirements. txt

[hjj-lmx]

System Info / 系統信息

Windows 11

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

Using cached deepspeed-0.16.2.tar.gz (1.4 MB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [50 lines of output]
test.c
LINK : fatal error LNK1181: 无法打开输入文件“aio.lib”
<3>WSL (12) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
<3>WSL (15) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
<3>WSL (18) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
<3>WSL (21) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
<3>WSL (24) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
<3>WSL (27) ERROR: CreateProcessCommon:559: execvpe(/bin/bash) failed: No such file or directory
test.c
LINK : fatal error LNK1181: 无法打开输入文件“cufile.lib”
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "C:\Users\lhcx\AppData\Local\Temp\pip-install-zvlda6cj\deepspeed_cde69103da1f45f7ab3eba525a4e5377\setup.py", line 200, in
ext_modules.append(builder.builder())
File "C:\Users\lhcx\AppData\Local\Temp\pip-install-zvlda6cj\deepspeed_cde69103da1f45f7ab3eba525a4e5377\op_builder\builder.py", line 720, in builder
extra_link_args=self.strip_empty_entries(self.extra_ldflags()))
File "C:\Users\lhcx\AppData\Local\Temp\pip-install-zvlda6cj\deepspeed_cde69103da1f45f7ab3eba525a4e5377\op_builder\inference_cutlass_builder.py", line 74,
in extra_ldflags
import dskernels
ModuleNotFoundError: No module named 'dskernels'
DS_BUILD_OPS=1
[WARNING] Skip pre-compile of incompatible async_io; One can disable async_io with DS_BUILD_AIO=0
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] cpu_adam requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_adam attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your sy
stem, will fall back to use -march=native and non-vectorized execution.
[WARNING] cpu_adam requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_adam attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your sy
stem, will fall back to use -march=native and non-vectorized execution.
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] cpu_adagrad requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_adagrad attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your
system, will fall back to use -march=native and non-vectorized execution.
[WARNING] cpu_adagrad requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_adagrad attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your
system, will fall back to use -march=native and non-vectorized execution.
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] cpu_lion requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_lion attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your sy
stem, will fall back to use -march=native and non-vectorized execution.
[WARNING] cpu_lion requires the 'lscpu' command, but it does not exist!
[WARNING] cpu_lion attempted to query 'lscpu' after failing to use py-cpuinfo to detect the CPU architecture. 'lscpu' does not appear to exist on your sy
stem, will fall back to use -march=native and non-vectorized execution.
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] Skip pre-compile of incompatible evoformer_attn; One can disable evoformer_attn with DS_BUILD_EVOFORMER_ATTN=0
[WARNING] Skip pre-compile of incompatible fp_quantizer; One can disable fp_quantizer with DS_BUILD_FP_QUANTIZER=0
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] Skip pre-compile of incompatible gds; One can disable gds with DS_BUILD_GDS=0
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
Installed CUDA version 12.3 does not match the version torch was compiled with 12.4 but since the APIs are compatible, accepting this combination
[WARNING] Filtered compute capabilities ['6.0', '6.1', '7.0']
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Expected behavior / 期待表现

success

[nitinmukesh]

DeepSpeed is the worst project, From Microsoft but not supported on Windows.
microsoft/DeepSpeed#6923

See if this helps
https://youtu.be/mYzjy7tDdAM?si=0-WJjqabgPHmVck9

[zRzRzRzRzRzRzR]

Deepspeed is not necessary. If you don't need to do fine-tuning, you don't need to install it. If you do need to fine-tune, you need to install it. You can refer to the solution provided by @nitinmukesh.