Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][experimental] Support for GPU tensors nested inside other data #19

Closed
wants to merge 2 commits into from

Conversation

stephanie-wang
Copy link
Owner

@stephanie-wang stephanie-wang commented May 16, 2024

Why are these changes needed?

Depends on ray-project#45332. Adds a nested TorchTensorType so that tensors stored inside a Python object can be passed via NCCL using the following syntax:

    with InputNode() as inp:
        dag = sender.send.bind(inp)
        dag = dag.with_contains_type_hint(TorchTensorType(transport="nccl"))
        dag = receiver.recv.bind(dag)

    compiled_dag = dag.experimental_compile()

with_contains_type_hint indicates that the value returned by that DAG node contains one or more torch.Tenors that should be transferred via NCCL. We implement this using a shared memory channel to pass CPU data, and the existing NCCL channel. If the TorchTensorType doesn't have a shape and dtype specified, we use an additional shared memory channel to pass metadata for the serialized tensors.

Related issue number

Checks

Sorry, something went wrong.

Your Name added 2 commits May 15, 2024 16:38
Signed-off-by: Your Name <[email protected]>
test
Signed-off-by: Your Name <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant