Int8 pipeline parallelism

[psinger]

I am trying to work with cuda streams for pipeline parallelism, i.e. executing different parts of a model at the same time on different gpus.
And with int4, float16, bfloat16 everything seems to work as expected.

However, with int8 there appears to be something blocking, and gpus execute sequentially.
As int4 works, I am wondering if anyone knows if there is some blocking operation in int8.

Thanks!