Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit 48089c1

Browse files
authoredOct 9, 2024··
feat: add the design implementation for the sd artifacts download [skip ci] (#3220)
1 parent 3b6e8a4 commit 48089c1

File tree

1 file changed

+73
-0
lines changed

1 file changed

+73
-0
lines changed
 

‎design/sd-download-artifacts.md

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# SD Download Zipped Artifacts
2+
3+
#### Context
4+
5+
The current [API endpoint](https://github.com/screwdriver-cd/screwdriver/blob/master/plugins/builds/artifacts/get.js) allows downloading artifacts for a single file. This is quite limited, as users often need to download multiple files simultaneously.
6+
7+
#### Proposal
8+
9+
###### Asynchronous Implementation (preferred for a foolproof solution)
10+
11+
```mermaid
12+
sequenceDiagram
13+
Client->>API: Download dir
14+
API->>DB: Save request
15+
API->>Client: Return 201 with a link to get download status
16+
Zipper->>DB: Consume request
17+
create participant Store
18+
Zipper->>Store: Stream files
19+
destroy Store
20+
Zipper->>Store: Close connection
21+
Zipper->>Zipper: zipping
22+
create participant S3
23+
Zipper->>S3: upload zipped file
24+
destroy S3
25+
S3->>Zipper: send the URL info
26+
destroy Zipper
27+
Zipper->>DB: update request with S3 URL
28+
Client->>API: Download dir status
29+
destroy DB
30+
API->>DB: Get download URL based on request ID
31+
API->>Client: Return 200 with S3 URL
32+
```
33+
34+
###### Async Implementation Flows
35+
36+
1. User requests to download a directory.
37+
2. The client sends a download directory request to the API.
38+
3. The API saves the request in the database.
39+
4. The API returns a 201 response to the client with a link to check the download status.
40+
5. The Zipper service consumes the request from the database.
41+
6. The Zipper service calls Store and streams the files from the store.
42+
7. The Zipper service starts zipping the streamed files.
43+
8. The Zipper service uploads the zipped file to S3.
44+
9. The S3 sends the URL information back to the Zipper service.
45+
10. The Zipper service updates the request in the database with the S3 URL.
46+
11. The client sends a request to check the download directory status to the API.
47+
12. The API retrieves the download URL based on the request ID from the database.
48+
13. The API returns a 200 response to the client with the S3 URL.
49+
50+
###### Synchronous Implementation (fast development time)
51+
52+
```mermaid
53+
sequenceDiagram
54+
Client->>API: Download dir
55+
loop Directory tree
56+
API->>Store: Stream files
57+
end
58+
API->>API: zipping
59+
API->>Client: return the zipped file
60+
```
61+
62+
###### Synchronous Implementation Flows
63+
64+
1. User requests to download a directory.
65+
2. The client sends a download directory request to the API.
66+
3. The API begins streaming the files from the store in a loop, recursively traversing the entire directory tree.
67+
4. The API zips the streamed files.
68+
5. The API returns the zipped file to the client.
69+
6. The client receives the zipped file.
70+
71+
#### Design Caveat
72+
73+
Depending on the number of files being streamed, there’s a potential risk of running into memory issues. To prevent this, we should implement a check to estimate the total download size, such as 1GB.

0 commit comments

Comments
 (0)
Please sign in to comment.