[Feature Request] s5cmd command support for COPY Persistent Data Storage

[turian]

Persistent Data Storage that is COPYéd can be quite slow to copy.

s5cmd is MUCH faster at uploading and downloading from S3 / R2 that s3cmd and aws-cli. Like MUCH faster: "For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively."

NOTE: s5cmd is only fast when it can parallelize into many downloads. It is of comparable speed if you are just downloading one big file.

[romilbhardwaj]

Thanks for the pointer @turian! We'll look into using s5cmd and integrate it if it improves perf without breaking user workflows.

cc @landscapepainter

[landscapepainter]

I will go ahead and benchmark for a comparison and see if it has all the necessary functionality for Skypilot as well. Thanks @turian !

[turian]

@landscapepainter @romilbhardwaj Awesome! It's a nifty little tool, because usually the most painful step when you spin up a machine is getting the data onto disk from blob storage. Total dream come true.

I'm crossing my fingers it's an easy integration from the skypilot team.

[landscapepainter]

Confirmed that s5cmd supports for symlink exclusion flag, --exclude option, md5 checksum for upload, and also supports for specifying profile with --profile/--credentials-file/--endpoint-url (for R2 support), but it lacks --include option yet, which is used in storage.py/get_file_sync_command for R2Store and S3Store.

Ran a quick benchmark and it is impressively quick compared to aws-cli:
1000 of 1MB files from local to s3 sync on a medium tier disk:

• s5cmd sync: 7.53s
• aws s3 sync with 10 threads(default): 20.73s

6 of 10GB files from local to s3 sync on a medium tier disk:

• s5cmd sync: 386.55s
• aws s3 sync with 10 threads(default): 612.96s

We may need to do more checks before we actually implement it, but this seems promising! @romilbhardwaj @turian

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.

[romilbhardwaj]

#2291 is fixing this

[turian]

Amazing! Looking forward to seeing it land

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This issue was closed because it has been stalled for 10 days with no activity.