struggle for ninja when running test.py

[dzyjjpy]

I met the issue and try to solve it but failed. Dose someone have experience?
I followed #33
#25

jiapy@adminroot:/workspace/tools/ninja/ninja-1.10.0$ source /home/jiapy/virtualEnv/py3.6torch1.1/bin/activate
(py3.6torch1.1) jiapy@adminroot:/workspace/tools/ninja/ninja-1.10.0$ ninja --version
1.10.0
(py3.6torch1.1) jiapy@adminroot:/workspace/tools/ninja/ninja-1.10.0$ ninja -v
ninja: no work to do.
(py3.6torch1.1) jiapy@adminroot:/workspace/tools/ninja/ninja-1.10.0$ sh /home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/test.sh
Traceback (most recent call last):
File "/home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 949, in _build_extension_module
check=True)
File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/test.py", line 25, in
import models
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/../lib/models/init.py", line 11, in
import models.seg_hrnet
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/../lib/models/seg_hrnet.py", line 22, in
from .sync_bn.inplace_abn.bn import InPlaceABNSync
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/../lib/models/sync_bn/init.py", line 1, in
from .inplace_abn import bn
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/../lib/models/sync_bn/inplace_abn/init.py", line 1, in
from .bn import ABN, InPlaceABN, InPlaceABNSync
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/tools/../lib/models/sync_bn/inplace_abn/bn.py", line 14, in
from functions import *
File "/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/functions.py", line 16, in
extra_cuda_cflags=["--expt-extended-lambda"])
File "/home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 644, in load
is_python_module)
File "/home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 813, in jit_compile
with_cuda=with_cuda)
File "/home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 866, in write_ninja_file_and_build
build_extension_module(name, build_directory, verbose)
File "/home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 962, in build_extension_module
raise RuntimeError(message)
RuntimeError: Error building extension 'inplace_abn': b"[1/3] :/usr/local/cuda-10.0:/usr/local/cuda-10.0/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/TH -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-10.0:/usr/local/cuda-10.0/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cuda.cu -o inplace_abn_cuda.cuda.o\nFAILED: inplace_abn_cuda.cuda.o \n:/usr/local/cuda-10.0:/usr/local/cuda-10.0/bin/nvcc -DTORCH_EXTENSION_NAME=inplace_abn -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/TH -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-10.0:/usr/local/cuda-10.0/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' --expt-extended-lambda -std=c++11 -c /home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cuda.cu -o inplace_abn_cuda.cuda.o\n/bin/sh: 1: :/usr/local/cuda-10.0:/usr/local/cuda-10.0/bin/nvcc: not found\n[2/3] c++ -MMD -MF inplace_abn_cpu.o.d -DTORCH_EXTENSION_NAME=inplace_abn -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/TH -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-10.0:/usr/local/cuda-10.0/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -O3 -c /home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp -o inplace_abn_cpu.o\nFAILED: inplace_abn_cpu.o \nc++ -MMD -MF inplace_abn_cpu.o.d -DTORCH_EXTENSION_NAME=inplace_abn -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/TH -isystem /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/include/THC -isystem :/usr/local/cuda-10.0:/usr/local/cuda-10.0/include -isystem /home/jiapy/virtualEnv/py3.6torch1.1/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -O3 -c /home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp -o inplace_abn_cpu.o\n/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp: In function \xe2\x80\x98std::vectorat::Tensor backward_cpu(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, bool, float)\xe2\x80\x99:\n/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp:82:34: error: could not convert \xe2\x80\x98z.at::Tensor::type()\xe2\x80\x99 from \xe2\x80\x98at::DeprecatedTypeProperties\xe2\x80\x99 to \xe2\x80\x98c10::IntArrayRef {aka c10::ArrayRef}\xe2\x80\x99\n auto dweight = at::empty(z.type(), {0});\n ^\n/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp:83:32: error: could not convert \xe2\x80\x98z.at::Tensor::type()\xe2\x80\x99 from \xe2\x80\x98at::DeprecatedTypeProperties\xe2\x80\x99 to \xe2\x80\x98c10::IntArrayRef {aka c10::ArrayRef}\xe2\x80\x99\n auto dbias = at::empty(z.type(), {0});\n ^\n/home/jiapy/workspace/segmentation/HRNet-Semantic-Segmentation/lib/models/sync_bn/inplace_abn/src/inplace_abn_cpu.cpp:89:29: error: could not convert \xe2\x80\x98{dx, dweight, dbias}\xe2\x80\x99 from \xe2\x80\x98\xe2\x80\x99 to \xe2\x80\x98std::vectorat::Tensor\xe2\x80\x99\n return {dx, dweight, dbias};\n ^\nninja: build stopped: subcommand failed.\n"

[dzyjjpy]

PS: I tried pytorch 1.0, 1.1, 1.2 on branch-1.1
my environment:
ubuntu:16.04
python:3.6
gcc: 5.4
cuda: 10.0

and I install re2c and ninja by source code.

[dzyjjpy]

some related issue on Github, FYI
mapillary/inplace_abn#104
ninja-build/ninja#1582
mapillary/inplace_abn#106
ninja-build/ninja#1633

[dzyjjpy]

@sunke123 @leoxiaobin @welleast

[uyoung-jeong]

From my experience, ninja-related problems are solved by using pytorch-v1.1 branch.

env:
ubuntu 18.04
python 3.6.9
pytorch 1.3.1

[songyuc]

Hi, @uyoung-jeong , I still met this problem today.

[tjzjp]

I met the same problem.
my env is:
python 3.6.10
pytorch 1.1.0
ninja 1.9.0
cuda10.0

[sunpeng981712364]

I met the same problem.
my env is:
python 3.7
pytorch 1.1.0
ninja 1.9.0
cuda10.0

[muyue1238]

changing line 956 of /home/jiapy/virtualEnv/py3.6torch1.1/lib/python3.6/site-packages/torch/utils/cpp_extension.py from
['ninja', '-v']
to
['ninja', '--version']
seems solved the problem.