index_cpu_to_all_gpus is very slow for the first time

[abhishekukmeesho]

I have a simple code where I load the index from a path and converting it to a GPU index

faiss_index = faiss.read_index('/path/to/specific/faiss.index')
faiss_index_gpu = faiss.index_cpu_to_all_gpus(faiss_index)

I'm using:

Python = 3.10
faiss-gpu==1.7.2
CUDA version (torch): 11.8
Size of the index ~= 800MB
Machine = g2-standard-8 = Nvidia L4 +  8vCPU + 32G CPU mem

The issue is, whenever I run it for the first time, it almost takes 20 minutes for the second line to execute(index_cpu_to_all_gpus). But if I load a new index later on and call the same function or even run the same code block the second time, it finishes within a couple of seconds.

So I wanted to understand why is this behaviour seen and if I am making any mistake. Any help is appreciated.

[Arcmoon-Hu]

same issue

[mdouze]

Maybe
https://github.com/facebookresearch/faiss/wiki/Troubleshooting#slow-gpu-intialization

[abhishekukmeesho]

I used the 1.7.3 version with wheel as explained here (kyamagu/faiss-wheels#54 (comment)) and it seems to have solved the issue. Now the load happens instantaneously.