specs,schedulers: add Role.mounts+BindMount and implement in docker,kubernetes,aws_batch schedulers

[d4l3k]

This adds a new Role option called mounts that allows for specifying different types of mounts on the worker. Currently the only supported mount is BindMount which does a simple bind from host to container. For more complex binds in the future we can easily extend mounts: List[BindMount] to mounts: List[Union[BindMount, PersistentVolumeClaimMount]] if necessary

This also adds in a new argument to dist.ddp

$ torchx run --scheduler local_docker --wait --log dist.ddp --script dist_app.py -j 1x1 --mounts type=bind,src=/tmp,dst=/tmp,readonly

To help with these parsing it there's a new parse_mounts method in specs. The format for these is:

type=bind,src=<src>,dst=<dst>[,readonly][,type=bind,...]

This follows the docker CLI format https://docs.docker.com/storage/bind-mounts/

Fixes #415

References

• https://docker-py.readthedocs.io/en/stable/api.html#docker.types.Mount
• https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
• https://docs.aws.amazon.com/batch/latest/APIReference/API_MountPoint.html

Test plan:

CI

Still need to add some more unit tests per scheduler (maybe component test?)

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[codecov]

Codecov Report

Merging #420 (5c982ab) into main (d1669d7) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #420      +/-   ##
==========================================
+ Coverage   94.23%   94.29%   +0.06%     
==========================================
  Files          67       67              
  Lines        3763     3805      +42     
==========================================
+ Hits         3546     3588      +42     
  Misses        217      217              

Impacted Files	Coverage Δ
torchx/schedulers/local_scheduler.py	92.95% <ø> (ø)
torchx/schedulers/ray_scheduler.py	93.93% <ø> (ø)
torchx/schedulers/slurm_scheduler.py	97.46% <ø> (ø)
torchx/specs/__init__.py	96.15% <ø> (ø)
torchx/components/dist.py	79.54% <100.00%> (ø)
torchx/schedulers/aws_batch_scheduler.py	83.33% <100.00%> (+0.53%)	⬆️
torchx/schedulers/docker_scheduler.py	96.19% <100.00%> (+0.06%)	⬆️
torchx/schedulers/kubernetes_scheduler.py	93.00% <100.00%> (+0.21%)	⬆️
torchx/specs/api.py	99.00% <100.00%> (+0.09%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1669d7...5c982ab. Read the comment docs.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.