Windows 2022 Runner - Cuda 12.5.x installer hangs

[heathhenley]

Recently cuda installs started to hang (or take so long they timeout at 1.5 hours so far - I didn't try yet with no timeout) on the github 2022 windows runner.

Everything was working until as recently as yesterday, so I suspect it's an update to the runner and not cuda-toolkit (we've had the version pinned).

Hoping you might be able point me in the right direction to start to figure out what's going on though - maybe there's a way I can log to see if there's an error other output on the runner that would give us a clue about it?

Here's a minimal action to reproduce and there's one running here: https://github.com/heathhenley/test_windows_action_cuda_install/actions/runs/13573647553 - as far as I can tell it's just hanging there on the installer call:

name: Install CUDA Windows
on:
  push:
    branches:
      - 'main'
      - 'master'
jobs:
  install-cuda:
    runs-on: windows-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4
      - name: Install cuda-toolkit
        uses: Jimver/[email protected]
        with:
          cuda: '12.5.0'

Maybe the answer is to open an issue on the runner image repo, but given that it's hanging (or taking significantly longer) with this action, I'm hoping you might have some tips.

Any help is appreciated, thanks for putting this action out there!

[Silverlan]

Same problem here.
It's just a guess, but I believe it has to do with this issue:
https://forums.developer.nvidia.com/t/stuck-installing-nsight-on-cuda-12-8-0-571-96-windows/323732

When trying to install CUDA on my PC, the installer also hangs during the NSight Visual Studio installation.

I suspect that there was a GitHub runner update that has updated Visual Studio to a newer version (since the problem doesn't seem to occur with older versions).
If that's the case, a possible fix/workaround could be to disable the installation of NSight Visual Studio in the action, but I'm not sure how to do that.

[heathhenley]

Good find that definitely seems like the culprit...

This script posted on another issue seems to install cuda no problem, maybe it will help someone: #253 (comment)

[N-Storm]

Issue that addresses this bug: microsoft/STL#5282

Quote from it:

(Update: This is a VS bug, DevCom-10841757, expected to be fixed in VS 2022 17.14 Preview 2.)

Not only 12.5.0 are affected, versions up to 12.8.0 are affected as well.

[andrey-khropov]

Same problem here. It's just a guess, but I believe it has to do with this issue: https://forums.developer.nvidia.com/t/stuck-installing-nsight-on-cuda-12-8-0-571-96-windows/323732

When trying to install CUDA on my PC, the installer also hangs during the NSight Visual Studio installation.

I suspect that there was a GitHub runner update that has updated Visual Studio to a newer version (since the problem doesn't seem to occur with older versions).

Yes. Visual Studio 2022 has been updated to 17.3.x version.

If that's the case, a possible fix/workaround could be to disable the installation of NSight Visual Studio in the action,

nVidia answered that you should disable two components during installation:

• “Nsight VSE”
• "Visual Studio Integration"

https://forums.developer.nvidia.com/t/nvidia-nsight-visual-studio-edition-2025-1-0-25002-setup-stuck-on-configuring-visual-studio-2022-settings-for-nsight-visual-studio-edition/323713/5

but I'm not sure how to do that.

This action has sub-packages parameter where you can select only some sub-packages to install.

I can confirm that the installation works with subpackages '["nvcc","cudart","thrust"]'

(catboost/catboost@147855b)

[N-Storm]

I can confirm that the installation works with subpackages '["nvcc","cudart","thrust"]'

Note that this way it doesn't adds $CUDA_PATH/bin to $PATH env. So if your build system doesn't discovers path to nvcc, simply calling nvcc won't work. Either add it to PATH or call it by full path.

[andrey-khropov]

I can confirm that the installation works with subpackages '["nvcc","cudart","thrust"]'

Note that this way it doesn't adds $CUDA_PATH/bin to $PATH env. So if your build system doesn't discovers path to nvcc, simply calling nvcc won't work. Either add it to PATH or call it by full path.

But it sets

    CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
    CUDA_PATH_V11_8: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
    CUDA_PATH_VX_Y: CUDA_PATH_V11_8

so you can add it to $PATH by yourself if necessary.

[N-Storm]

@andrey-khropov, yes, that's exactly that I've meant. Just a reminder that you have to add it on your own. Because previously visual_studio_integration package (installed by default) added $CUDA_PATH/bin on install.