|
| 1 | +# SD Download Zipped Artifacts |
| 2 | + |
| 3 | +#### Context |
| 4 | + |
| 5 | +The current [API endpoint](https://github.com/screwdriver-cd/screwdriver/blob/master/plugins/builds/artifacts/get.js) allows downloading artifacts for a single file. This is quite limited, as users often need to download multiple files simultaneously. |
| 6 | + |
| 7 | +#### Proposal |
| 8 | + |
| 9 | +###### Asynchronous Implementation (preferred for a foolproof solution) |
| 10 | + |
| 11 | +```mermaid |
| 12 | +sequenceDiagram |
| 13 | + Client->>API: Download dir |
| 14 | + API->>DB: Save request |
| 15 | + API->>Client: Return 201 with a link to get download status |
| 16 | + Zipper->>DB: Consume request |
| 17 | + create participant Store |
| 18 | + Zipper->>Store: Stream files |
| 19 | + destroy Store |
| 20 | + Zipper->>Store: Close connection |
| 21 | + Zipper->>Zipper: zipping |
| 22 | + create participant S3 |
| 23 | + Zipper->>S3: upload zipped file |
| 24 | + destroy S3 |
| 25 | + S3->>Zipper: send the URL info |
| 26 | + destroy Zipper |
| 27 | + Zipper->>DB: update request with S3 URL |
| 28 | + Client->>API: Download dir status |
| 29 | + destroy DB |
| 30 | + API->>DB: Get download URL based on request ID |
| 31 | + API->>Client: Return 200 with S3 URL |
| 32 | +``` |
| 33 | + |
| 34 | +###### Async Implementation Flows |
| 35 | + |
| 36 | +1. User requests to download a directory. |
| 37 | +2. The client sends a download directory request to the API. |
| 38 | +3. The API saves the request in the database. |
| 39 | +4. The API returns a 201 response to the client with a link to check the download status. |
| 40 | +5. The Zipper service consumes the request from the database. |
| 41 | +6. The Zipper service calls Store and streams the files from the store. |
| 42 | +7. The Zipper service starts zipping the streamed files. |
| 43 | +8. The Zipper service uploads the zipped file to S3. |
| 44 | +9. The S3 sends the URL information back to the Zipper service. |
| 45 | +10. The Zipper service updates the request in the database with the S3 URL. |
| 46 | +11. The client sends a request to check the download directory status to the API. |
| 47 | +12. The API retrieves the download URL based on the request ID from the database. |
| 48 | +13. The API returns a 200 response to the client with the S3 URL. |
| 49 | + |
| 50 | +###### Synchronous Implementation (fast development time) |
| 51 | + |
| 52 | +```mermaid |
| 53 | +sequenceDiagram |
| 54 | + Client->>API: Download dir |
| 55 | + loop Directory tree |
| 56 | + API->>Store: Stream files |
| 57 | + end |
| 58 | + API->>API: zipping |
| 59 | + API->>Client: return the zipped file |
| 60 | +``` |
| 61 | + |
| 62 | +###### Synchronous Implementation Flows |
| 63 | + |
| 64 | +1. User requests to download a directory. |
| 65 | +2. The client sends a download directory request to the API. |
| 66 | +3. The API begins streaming the files from the store in a loop, recursively traversing the entire directory tree. |
| 67 | +4. The API zips the streamed files. |
| 68 | +5. The API returns the zipped file to the client. |
| 69 | +6. The client receives the zipped file. |
| 70 | + |
| 71 | +#### Design Caveat |
| 72 | + |
| 73 | +Depending on the number of files being streamed, there’s a potential risk of running into memory issues. To prevent this, we should implement a check to estimate the total download size, such as 1GB. |
0 commit comments