[Core][experimental] Failure detection doesn't happen properly when the worker fails with RuntimeError

[rkooo567]

What happened + What you expected to happen

From OSS vllm, remove

        # for worker, (node_id, _) in zip(self.workers, worker_node_and_gpu_ids):
            # worker.set_cuda_visible_devices.remote(node_gpus[node_id])

The execute_method fails with

Error Type: TASK_EXECUTION_EXCEPTION23py", line 69, in init_model4    torch.cuda.set_device(self.device)5  File "/home/ray/anaconda3/lib/python3.9/site-packages/torch/cuda/__init__.py", line 404, in set_device6    torch._C._cuda_setDevice(device)7RuntimeError: CUDA error: invalid device ordinal8CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.9For debugging consider passing CUDA_LAUNCH_BLOCKING=1.10Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

but the driver hangs.

Versions / Dependencies

master

Reproduction script

vllm-project/vllm#2462

And comment out code ^

Issue Severity

None

[anyscalesam]

@rkooo567 is this bug still valid?

[rkooo567]

verfiied I cannot repro