Add a device parameter to RemoteModule #44254

wayi1 · 2020-09-05T23:48:58Z

Summary:
Add a device parameter to RemoteModule, so it can be placed on any device
and not just CPU.

Original PR issue: RemoteModule enhancements #40550

Test Plan: buck test test/distributed/rpc:process_group_agent -- RemoteModule

Differential Revision: D23483803

facebook-github-bot · 2020-09-05T23:49:13Z

This pull request was exported from Phabricator. Differential Revision: D23483803

dr-ci · 2020-09-06T00:43:54Z

💊 CI failures summary and remediations

As of commit 3e64802 (more details on the Dr. CI page):

2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_6_gcc5_4_test (1/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 18 07:34:42 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future

Sep 18 07:34:42 At: 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:42  
Sep 18 07:34:42 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 07:34:42  
Sep 18 07:34:42 At: 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:42  
Sep 18 07:34:42 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 07:34:42  
Sep 18 07:34:42 At: 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:42   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:42  
Sep 18 07:34:42 ok (1.638s) 
Sep 18 07:34:43   test_return_future_remote (__main__.TensorPipeRpcTestWithSpawn) ... [W tensorpipe_agent.cpp:576] RPC agent for worker3 encountered error when reading incoming request from worker0: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 07:34:43 [W tensorpipe_agent.cpp:576] RPC agent for worker0 encountered error when reading incoming request from worker1: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 07:34:44 ok (1.648s) 
Sep 18 07:34:45   test_return_local_rrefs (__main__.TensorPipeRpcTestWithSpawn) ... [W tensorpipe_agent.cpp:576] RPC agent for worker3 encountered error when reading incoming request from worker0: EOF: end of file (this is expected to happen during shutdown)

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (2/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Sep 18 07:34:16 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future

Sep 18 07:34:16 At: 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:16  
Sep 18 07:34:16 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 07:34:16  
Sep 18 07:34:16 At: 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:16  
Sep 18 07:34:16 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 18 07:34:16  
Sep 18 07:34:16 At: 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 18 07:34:16   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 18 07:34:16  
Sep 18 07:34:16 [W tensorpipe_agent.cpp:576] RPC agent for worker3 encountered error when reading incoming request from worker0: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 07:34:16 [W tensorpipe_agent.cpp:576] RPC agent for worker1 encountered error when reading incoming request from worker0: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 07:34:16 ok (1.536s) 
Sep 18 07:34:17   test_return_future_remote (__main__.TensorPipeRpcTestWithSpawn) ... [W tensorpipe_agent.cpp:576] RPC agent for worker3 encountered error when reading incoming request from worker2: EOF: end of file (this is expected to happen during shutdown) 
Sep 18 07:34:17 [W tensorpipe_agent.cpp:576] RPC agent for worker0 encountered error when reading incoming request from worker2: EOF: end of file (this is expected to happen during shutdown)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 36 times.

codecov · 2020-09-06T02:55:34Z

Codecov Report

❗ No coverage uploaded for pull request base (master@ed862d3). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #44254   +/-   ##
=========================================
  Coverage          ?   67.95%           
=========================================
  Files             ?      384           
  Lines             ?    49597           
  Branches          ?        0           
=========================================
  Hits              ?    33702           
  Misses            ?    15895           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed862d3...3049dd6. Read the comment docs.

Pull Request resolved: #44254 Add a device parameter to RemoteModule, so it can be placed on any device and not just CPU. Original PR issue: RemoteModule enhancements #40550 Differential Revision: [D23483803](https://our.internmc.facebook.com/intern/diff/D23483803/) [ghstack-poisoned]

facebook-github-bot · 2020-09-07T22:56:05Z

This pull request was exported from Phabricator. Differential Revision: D23483803

torch/distributed/nn/api/remote_module.py

torch/testing/_internal/distributed/nn/api/remote_module_test.py

torch/distributed/nn/api/remote_module.py

facebook-github-bot · 2020-09-11T07:30:07Z

This pull request was exported from Phabricator. Differential Revision: D23483803

facebook-github-bot · 2020-09-11T21:58:49Z

This pull request was exported from Phabricator. Differential Revision: D23483803

facebook-github-bot · 2020-09-11T22:04:51Z

This pull request was exported from Phabricator. Differential Revision: D23483803

facebook-github-bot · 2020-09-11T22:34:21Z

This pull request was exported from Phabricator. Differential Revision: D23483803

torch/distributed/nn/api/remote_module.py

facebook-github-bot · 2020-09-15T03:52:08Z

This pull request was exported from Phabricator. Differential Revision: D23483803

torch/distributed/nn/api/remote_module.py

facebook-github-bot · 2020-09-18T05:14:32Z

This pull request was exported from Phabricator. Differential Revision: D23483803

facebook-github-bot · 2020-09-18T05:28:06Z

This pull request was exported from Phabricator. Differential Revision: D23483803

Summary: Pull Request resolved: #44254 Add a device parameter to RemoteModule, so it can be placed on any device and not just CPU. Original PR issue: RemoteModule enhancements #40550 Test Plan: buck test test/distributed/rpc:process_group_agent -- RemoteModule Reviewed By: pritamdamania87 Differential Revision: D23483803 fbshipit-source-id: d0459f94906b1c4df1fee3cda981c5249c78b842

facebook-github-bot · 2020-09-18T05:44:46Z

This pull request was exported from Phabricator. Differential Revision: D23483803

facebook-github-bot · 2020-09-18T18:15:02Z

This pull request has been merged in c68cc78.

wayi1 requested review from apaszke, mrshenli, pietern, pritamdamania87 and zhaojuanmao as code owners September 5, 2020 23:48

facebook-github-bot added the fb-exported label Sep 5, 2020

wayi1 removed request for pietern, apaszke, mrshenli and zhaojuanmao September 5, 2020 23:51

wayi1 self-assigned this Sep 5, 2020

wayi1 requested a review from mrshenli September 8, 2020 22:36

pritamdamania87 suggested changes Sep 10, 2020

View reviewed changes

wayi1 requested a review from pritamdamania87 September 11, 2020 07:24

pritamdamania87 reviewed Sep 15, 2020

View reviewed changes

torch/distributed/nn/api/remote_module.py Outdated Show resolved Hide resolved

torch/distributed/nn/api/remote_module.py Outdated Show resolved Hide resolved

torch/distributed/nn/api/remote_module.py Outdated Show resolved Hide resolved

wayi1 requested a review from pritamdamania87 September 15, 2020 03:52

pritamdamania87 approved these changes Sep 15, 2020

View reviewed changes

pritamdamania87 reviewed Sep 16, 2020

View reviewed changes

torch/distributed/nn/api/remote_module.py Outdated Show resolved Hide resolved

wayi1 requested a review from rohan-varma as a code owner September 18, 2020 05:14

facebook-github-bot closed this in c68cc78 Sep 18, 2020

facebook-github-bot added the merged label Sep 18, 2020

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a device parameter to RemoteModule #44254

Add a device parameter to RemoteModule #44254

wayi1 commented Sep 5, 2020

facebook-github-bot commented Sep 5, 2020

dr-ci bot commented Sep 6, 2020 •

edited

Loading

codecov bot commented Sep 6, 2020 •

edited

Loading

facebook-github-bot commented Sep 7, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 15, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

Add a device parameter to RemoteModule #44254

Add a device parameter to RemoteModule #44254

Conversation

wayi1 commented Sep 5, 2020

facebook-github-bot commented Sep 5, 2020

dr-ci bot commented Sep 6, 2020 • edited Loading

💊 CI failures summary and remediations

🕵️ 2 new failures recognized by patterns

pytorch_linux_xenial_py3_6_gcc5_4_test (1/2)

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (2/2)

codecov bot commented Sep 6, 2020 • edited Loading

Codecov Report

facebook-github-bot commented Sep 7, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 11, 2020

facebook-github-bot commented Sep 15, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

facebook-github-bot commented Sep 18, 2020

dr-ci bot commented Sep 6, 2020 •

edited

Loading

codecov bot commented Sep 6, 2020 •

edited

Loading