Int8 pipeline parallelism

[psinger]

I am trying to work with cuda streams for pipeline parallelism, i.e. executing different parts of a model at the same time on different gpus.
And with int4, float16, bfloat16 everything seems to work as expected.

However, with int8 there appears to be something blocking, and gpus execute sequentially.
As int4 works, I am wondering if anyone knows if there is some blocking operation in int8.

Thanks!

[TimDettmers]

For int4 you are also using bitsandbytes code or is this only for int8? There are some operations on bitsandbytes that forces the cuda device before c-calls because this sometimes introduced bugs. It might be that this is causing your problems.

This behavior was changed in 0.45. Can you check your bitsandbytes version and see if you still have this problem with the newer version?

[matthewdouglas]

One further thing to note is that int8 has a host-device synchronization that is forced when decomposing the problem into separate int8 and fp16 matmuls. Using threshold=0.0 should avoid that, and will be faster in general, at the potential cost of accuracy.