vllm development does not work for tensor-parallel > 1 #2619
Comments
|
nvidia device info: lroberts@GPU77B9:~/llm_quantization$ nvidia-smi
Fri Jan 26 22:08:29 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:07:00.0 Off | 0 |
| N/A 34C P0 74W / 400W| 62027MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:0A:00.0 Off | 0 |
| N/A 31C P0 65W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM4-80GB On | 00000000:47:00.0 Off | 0 |
| N/A 32C P0 64W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM4-80GB On | 00000000:4D:00.0 Off | 0 |
| N/A 35C P0 68W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 4 NVIDIA A100-SXM4-80GB On | 00000000:87:00.0 Off | 0 |
| N/A 36C P0 68W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 5 NVIDIA A100-SXM4-80GB On | 00000000:8D:00.0 Off | 0 |
| N/A 33C P0 69W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 6 NVIDIA A100-SXM4-80GB On | 00000000:C7:00.0 Off | 0 |
| N/A 32C P0 66W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 7 NVIDIA A100-SXM4-80GB On | 00000000:CA:00.0 Off | 0 |
| N/A 35C P0 67W / 400W| 3MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 3727132 C python3.10 62014MiB |
+---------------------------------------------------------------------------------------+ |
|
also meet |
|
Looks like this is fixed in Ray 2.9 ray-project/ray#41913 (comment). Try upgrading Ray? We will make sure to lower bound the Ray version as well. |
I have tried
|
|
That seems to be a different issue, please open another ticket and I can try reproducing it. Update, it did try to reproduce it. With the latest main branch: |
as a work around I think you could pass the flag |
|
I believe #2642 might fix "resource already mapped", please try with latest main, sorry about the back and forth. |
|
@simon-mo This works for me: some stdout: also works for tensor-parallel 8 (all the ones on this machine) specs on A100 machine I'm using: python env: lroberts@GPU77B9:~/update-vllm-env/vllm-source/vllm$ python -c "import vllm, ray, torch, pydantic; print(vllm.__version__); print(ray.__version__); print(torch.__version__); print(pydantic.__version__)"
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (2.1.0) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
0.2.7
2.9.1
2.1.2+cu121
2.6.0
lroberts@GPU77B9:~/update-vllm-env/vllm-source/vllm$ git log -n 2
commit ea8489fce266d69f2fbe314c1385956b1a342e12 (HEAD -> main, origin/main, origin/HEAD)
Author: Rasmus Larsen <[email protected]>
Date: Mon Jan 29 19:52:31 2024 +0100
ROCm: Allow setting compilation target (#2581)
commit 1b20639a43e811f4469e3cfa543cf280d0d76265
Author: Hanzhi Zhou <[email protected]>
Date: Tue Jan 30 02:46:29 2024 +0800
No repeated IPC open (#2642)I don't think the inplace error -> #2620 is resolved though. I still see that one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a local dev build on commit
and I have some local code that is a thin wrapper around LLM class
If i run this with
tensor-parallel == 2I get the following:however,
tensor-parallel == 1works fine:with response:
{"choices":[{"finish_reason":"stop","index":0,"message":{"content":" There are many reasons why someone might consider higher education. Here are a few:\n\n1. To gain knowledge and skills: Higher education provides students with the opportunity to learn new knowledge and skills that can be applied in their future careers.\n2. To prepare for a career: Many people choose to pursue higher education because it is a way to prepare for a specific career. For example, a student may choose to study business because they want to work in the field.\n3. To gain a competitive edge: Higher education can provide students with a competitive edge in the job market. Many employers require a degree from a reputable institution, and having one can make a candidate more attractive to potential employers.\n4. To develop critical thinking and problem-solving skills: Higher education provides students with the opportunity to develop their critical thinking and problem-solving skills.\n5. To gain a sense of community: Higher education provides students with the opportunity to connect with other students and faculty members, which can help to create a sense of community.\n6. To gain a sense of purpose: Higher education can provide students with a sense of purpose and direction in life.\n7. To gain a sense of accomplishment: Higher education can provide students with a sense of accomplishment and pride in their achievements.\n8. To gain a sense of personal growth: Higher education can provide students with the opportunity to grow and develop as individuals.\n9. To gain a sense of independence: Higher education can provide students with the opportunity to become independent and self-sufficient.\n10. To gain a sense of fulfillment: Higher education can provide students with a sense of fulfillment and satisfaction in their lives.\n\nOverall, higher education can provide students with a wide range of benefits, including the opportunity to gain knowledge and skills, prepare for a career, gain a competitive edge, develop critical thinking and problem-solving skills, gain a sense of community, gain a sense of purpose, gain a sense of accomplishment, gain a sense of personal growth, gain a sense of independence, and gain a sense of fulfillment.","role":"assistant"}}],"created":1706306706,"id":"llama-2-7b-chat-hf","object":"chat.completion","usage":{"completion_tokens":457,"prompt_tokens":49,"total_tokens":506}}the error in logs from ray indicates some serialization
relevant details about env:
It seems there a known fix or workaround here -> ray-project/ray#41913 (comment)
but it seems that pydantic version 2 is necessary for openai testing
vllm/requirements.txt
Line 11 in 3a0e1fc
is there a suggested workaround or should I manually downgrade pydantic to version lower than 2.0.0?
The text was updated successfully, but these errors were encountered: