Skip to content

Commit bcea90a

Browse files
sdaberdakunineinchnick
sdaberdaku
authored and
nineinchnick
committed
Add option to enable worker graceful shutdown
1 parent 1956935 commit bcea90a

11 files changed

+236
-13
lines changed

‎charts/trino/README.md

+13-2
Original file line numberDiff line numberDiff line change
@@ -503,9 +503,11 @@ Fast distributed SQL query engine for big data analytics that helps you explore
503503

504504
Allows mounting additional Trino configuration files from Kubernetes secrets on the coordinator node.
505505
Example:
506+
```yaml
506507
- name: sample-secret
507508
secretName: sample-secret
508509
path: /secrets/sample.json
510+
```
509511
* `worker.jvm.maxHeapSize` - string, default: `"8G"`
510512
* `worker.jvm.gcMethod.type` - string, default: `"UseG1GC"`
511513
* `worker.jvm.gcMethod.g1.heapRegionSize` - string, default: `"32M"`
@@ -559,12 +561,21 @@ Fast distributed SQL query engine for big data analytics that helps you explore
559561
```
560562
* `worker.lifecycle` - object, default: `{}`
561563

562-
To enable [graceful shutdown](https://trino.io/docs/current/admin/graceful-shutdown.html), define a lifecycle preStop like bellow, Set the `terminationGracePeriodSeconds` to a value greater than or equal to the configured `shutdown.grace-period`. Configure `shutdown.grace-period` in `additionalConfigProperties` as `shutdown.grace-period=2m` (default is 2 minutes). Also configure `accessControl` because the `default` system access control does not allow graceful shutdowns.
564+
Worker container [lifecycle events](https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/) Setting `worker.lifecycle` conflicts with `worker.gracefulShutdown`.
563565
Example:
564566
```yaml
565567
preStop:
566568
exec:
567-
command: ["/bin/sh", "-c", "curl -v -X PUT -d '\"SHUTTING_DOWN\"' -H \"Content-type: application/json\" http://localhost:8081/v1/info/state"]
569+
command: ["/bin/sh", "-c", "sleep 120"]
570+
```
571+
* `worker.gracefulShutdown` - object, default: `{"enabled":false,"gracePeriodSeconds":120}`
572+
573+
Configure [graceful shutdown](https://trino.io/docs/current/admin/graceful-shutdown.html) in order to ensure that workers terminate without affecting running queries, given a sufficient grace period. When enabled, the value of `worker.terminationGracePeriodSeconds` must be at least two times greater than the configured `gracePeriodSeconds`. Enabling `worker.gracefulShutdown` conflicts with `worker.lifecycle`. When a custom `worker.lifecycle` configuration needs to be used, graceful shutdown must be configured manually.
574+
Example:
575+
```yaml
576+
gracefulShutdown:
577+
enabled: true
578+
gracePeriodSeconds: 120
568579
```
569580
* `worker.terminationGracePeriodSeconds` - int, default: `30`
570581
* `worker.nodeSelector` - object, default: `{}`

‎charts/trino/templates/configmap-access-control.yaml ‎charts/trino/templates/configmap-access-control-coordinator.yaml

File renamed without changes.

‎charts/trino/templates/configmap-access-control-worker.yaml

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{{- if .Values.worker.gracefulShutdown.enabled }}
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: {{ template "trino.fullname" . }}-access-control-volume-worker
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "trino.labels" . | nindent 4 }}
9+
app.kubernetes.io/component: worker
10+
data:
11+
graceful-shutdown-rules.json: >-
12+
{
13+
"system_information": [
14+
{
15+
"allow": [
16+
"write"
17+
],
18+
"user": "admin"
19+
}
20+
]
21+
}
22+
{{- end }}

‎charts/trino/templates/configmap-coordinator.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,9 @@ data:
7777
jmx.rmiregistry.port={{- $coordinatorJmx.registryPort }}
7878
jmx.rmiserver.port={{- $coordinatorJmx.serverPort }}
7979
{{- end }}
80+
{{- if .Values.worker.gracefulShutdown.enabled }}
81+
shutdown.grace-period={{- .Values.worker.gracefulShutdown.gracePeriodSeconds -}}s
82+
{{- end }}
8083
{{- if .Values.server.coordinatorExtraConfig }}
8184
{{- .Values.server.coordinatorExtraConfig | nindent 4 }}
8285
{{- end }}

‎charts/trino/templates/configmap-worker.yaml

+9
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,19 @@ data:
6565
jmx.rmiregistry.port={{- $workerJmx.registryPort }}
6666
jmx.rmiserver.port={{- $workerJmx.serverPort }}
6767
{{- end }}
68+
{{- if .Values.worker.gracefulShutdown.enabled }}
69+
shutdown.grace-period={{- .Values.worker.gracefulShutdown.gracePeriodSeconds -}}s
70+
{{- end }}
6871
{{- if .Values.server.workerExtraConfig }}
6972
{{- .Values.server.workerExtraConfig | nindent 4 }}
7073
{{- end }}
7174
75+
{{- if .Values.worker.gracefulShutdown.enabled }}
76+
access-control.properties: |
77+
access-control.name=file
78+
security.config-file={{ .Values.server.config.path }}/access-control/graceful-shutdown-rules.json
79+
{{- end }}
80+
7281
{{- if .Values.server.exchangeManager }}
7382
exchange-manager.properties: |
7483
exchange-manager.name={{ .Values.server.exchangeManager.name }}

‎charts/trino/templates/deployment-coordinator.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ spec:
1919
metadata:
2020
annotations:
2121
{{- if and (eq .Values.accessControl.type "configmap") (not .Values.accessControl.refreshPeriod) }}
22-
checksum/access-control-config: {{ include (print $.Template.BasePath "/configmap-access-control.yaml") . | sha256sum }}
22+
checksum/access-control-config: {{ include (print $.Template.BasePath "/configmap-access-control-coordinator.yaml") . | sha256sum }}
2323
{{- end }}
2424
{{- if or .Values.catalogs .Values.additionalCatalogs }}
2525
checksum/catalog-config: {{ include (print $.Template.BasePath "/configmap-catalog.yaml") . | sha256sum }}

‎charts/trino/templates/deployment-worker.yaml

+33
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ spec:
2626
checksum/catalog-config: {{ include (print $.Template.BasePath "/configmap-catalog.yaml") . | sha256sum }}
2727
{{- end }}
2828
checksum/worker-config: {{ include (print $.Template.BasePath "/configmap-worker.yaml") . | sha256sum }}
29+
{{- if .Values.worker.gracefulShutdown.enabled }}
30+
checksum/access-control-config: {{ include (print $.Template.BasePath "/configmap-access-control-worker.yaml") . | sha256sum }}
31+
{{- end }}
2932
{{- if .Values.worker.annotations }}
3033
{{- tpl (toYaml .Values.worker.annotations) . | nindent 8 }}
3134
{{- end }}
@@ -61,6 +64,11 @@ spec:
6164
configMap:
6265
name: {{ template "trino.fullname" . }}-jmx-exporter-config-worker
6366
{{- end }}
67+
{{- if .Values.worker.gracefulShutdown.enabled }}
68+
- name: access-control-volume
69+
configMap:
70+
name: {{ template "trino.fullname" . }}-access-control-volume-worker
71+
{{- end }}
6472
{{- range .Values.configMounts }}
6573
- name: {{ .name }}
6674
configMap:
@@ -92,7 +100,11 @@ spec:
92100
imagePullSecrets:
93101
{{- toYaml .Values.imagePullSecrets | nindent 8 }}
94102
{{- end }}
103+
{{- if and .Values.worker.gracefulShutdown.enabled (gt (mulf 2.0 .Values.worker.gracefulShutdown.gracePeriodSeconds) .Values.worker.terminationGracePeriodSeconds) }}
104+
{{- fail "The user must set the `worker.terminationGracePeriodSeconds` to a value of at least two times the configured `gracePeriodSeconds`." }}
105+
{{- else }}
95106
terminationGracePeriodSeconds: {{ .Values.worker.terminationGracePeriodSeconds }}
107+
{{- end }}
96108
containers:
97109
- name: {{ .Chart.Name }}-worker
98110
image: {{ include "trino.image" . }}
@@ -112,6 +124,10 @@ spec:
112124
{{- end }}
113125
- mountPath: {{ .Values.kafka.mountPath }}
114126
name: schemas-volume
127+
{{- if .Values.worker.gracefulShutdown.enabled }}
128+
- mountPath: {{ .Values.server.config.path }}/access-control
129+
name: access-control-volume
130+
{{- end }}
115131
{{- range .Values.configMounts }}
116132
- name: {{ .name }}
117133
mountPath: {{ .path }}
@@ -166,7 +182,24 @@ spec:
166182
failureThreshold: {{ .Values.worker.readinessProbe.failureThreshold | default 6 }}
167183
successThreshold: {{ .Values.worker.readinessProbe.successThreshold | default 1 }}
168184
lifecycle:
185+
{{- if .Values.worker.lifecycle }}
186+
{{- if .Values.worker.gracefulShutdown.enabled }}
187+
{{- fail "The `worker.lifecycle` configuration conflicts with `worker.gracefulShutdown`. Either disable `worker.gracefulShutdown` and apply the related configurations manually, or remove `worker.lifecycle`." }}
188+
{{- end }}
169189
{{- toYaml .Values.worker.lifecycle | nindent 12 }}
190+
{{- else if .Values.worker.gracefulShutdown.enabled }}
191+
preStop:
192+
exec:
193+
command:
194+
- /bin/sh
195+
- -c
196+
- >-
197+
curl -v -X PUT
198+
-d '"SHUTTING_DOWN"'
199+
-H 'Content-type: application/json'
200+
-H 'X-Trino-User: admin'
201+
http://localhost:{{- .Values.service.port -}}/v1/info/state
202+
{{- end }}
170203
resources:
171204
{{- toYaml .Values.worker.resources | nindent 12 }}
172205
{{- if $workerJmx.exporter.enabled }}

‎charts/trino/templates/tests/test-graceful-shutdown.yaml

+120
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
{{- if .Values.worker.gracefulShutdown.enabled }}
2+
apiVersion: rbac.authorization.k8s.io/v1
3+
kind: Role
4+
metadata:
5+
name: {{ include "trino.fullname" . }}-pod-manager
6+
namespace: {{ .Release.Namespace }}
7+
labels:
8+
{{- include "trino.labels" . | nindent 4 }}
9+
app.kubernetes.io/component: test
10+
test: graceful-shutdown
11+
annotations:
12+
"helm.sh/hook": test
13+
"helm.sh/hook-weight": "0"
14+
"helm.sh/hook-delete-policy": hook-succeeded
15+
rules:
16+
- apiGroups: [ "" ]
17+
resources: [ "pods" ]
18+
verbs: [ "get", "list", "delete" ]
19+
- apiGroups: [ "" ]
20+
resources: [ "pods/log" ]
21+
verbs: [ "get" ]
22+
---
23+
apiVersion: v1
24+
kind: ServiceAccount
25+
metadata:
26+
name: {{ include "trino.fullname" . }}-pod-manager-sa
27+
namespace: {{ .Release.Namespace }}
28+
labels:
29+
{{- include "trino.labels" . | nindent 4 }}
30+
app.kubernetes.io/component: test
31+
test: graceful-shutdown
32+
annotations:
33+
"helm.sh/hook": test
34+
"helm.sh/hook-weight": "0"
35+
"helm.sh/hook-delete-policy": hook-succeeded
36+
---
37+
apiVersion: rbac.authorization.k8s.io/v1
38+
kind: RoleBinding
39+
metadata:
40+
name: {{ include "trino.fullname" . }}-pod-manager-binding
41+
namespace: {{ .Release.Namespace }}
42+
labels:
43+
{{- include "trino.labels" . | nindent 4 }}
44+
app.kubernetes.io/component: test
45+
test: graceful-shutdown
46+
annotations:
47+
"helm.sh/hook": test
48+
"helm.sh/hook-weight": "1"
49+
"helm.sh/hook-delete-policy": hook-succeeded
50+
subjects:
51+
- kind: ServiceAccount
52+
name: {{ include "trino.fullname" . }}-pod-manager-sa
53+
namespace: {{ .Release.Namespace }}
54+
roleRef:
55+
kind: Role
56+
name: {{ include "trino.fullname" . }}-pod-manager
57+
apiGroup: rbac.authorization.k8s.io
58+
---
59+
apiVersion: v1
60+
kind: Pod
61+
metadata:
62+
name: {{ include "trino.fullname" . }}-test-graceful-shutdown
63+
labels:
64+
{{- include "trino.labels" . | nindent 4 }}
65+
app.kubernetes.io/component: test
66+
test: graceful-shutdown
67+
annotations:
68+
"helm.sh/hook": test
69+
"helm.sh/hook-weight": "2"
70+
"helm.sh/hook-delete-policy": hook-succeeded
71+
spec:
72+
serviceAccountName: {{ include "trino.fullname" . }}-pod-manager-sa
73+
initContainers:
74+
- name: get-worker-pod
75+
image: bitnami/kubectl:latest
76+
command: [ "sh", "-c" ]
77+
args:
78+
- >-
79+
kubectl get pods
80+
--selector="app.kubernetes.io/name={{ include "trino.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=worker"
81+
--output=jsonpath="{.items[0].metadata.name}"
82+
--namespace={{ .Release.Namespace }}
83+
> /pods/worker-pod.txt
84+
volumeMounts:
85+
- mountPath: /pods
86+
name: worker-pod
87+
containers:
88+
- name: check-logs
89+
image: bitnami/kubectl:latest
90+
command: [ "sh", "-c" ]
91+
args:
92+
- >-
93+
WORKER_POD=$(cat /pods/worker-pod.txt) &&
94+
kubectl logs ${WORKER_POD}
95+
--follow
96+
--container=trino-worker
97+
--namespace={{ .Release.Namespace }}
98+
| grep --max-count=1 "Shutdown requested"
99+
volumeMounts:
100+
- mountPath: /pods
101+
name: worker-pod
102+
- name: trigger-graceful-shutdown
103+
image: bitnami/kubectl:latest
104+
command: [ "sh", "-c" ]
105+
args:
106+
- >-
107+
sleep 5 &&
108+
WORKER_POD=$(cat /pods/worker-pod.txt) &&
109+
kubectl delete pod
110+
${WORKER_POD}
111+
--namespace={{ .Release.Namespace }}
112+
volumeMounts:
113+
- mountPath: /pods
114+
name: worker-pod
115+
restartPolicy: Never
116+
volumes:
117+
- name: worker-pod
118+
emptyDir: {}
119+
120+
{{- end }}

‎charts/trino/values.yaml

+27-9
Original file line numberDiff line numberDiff line change
@@ -586,9 +586,11 @@ coordinator:
586586
# files from Kubernetes secrets on the coordinator node.
587587
# @raw
588588
# Example:
589+
# ```yaml
589590
# - name: sample-secret
590591
# secretName: sample-secret
591592
# path: /secrets/sample.json
593+
# ```
592594

593595
worker:
594596
jvm:
@@ -661,20 +663,36 @@ worker:
661663
# ```
662664

663665
lifecycle: {}
664-
# worker.lifecycle -- To enable [graceful
665-
# shutdown](https://trino.io/docs/current/admin/graceful-shutdown.html),
666-
# define a lifecycle preStop like bellow, Set the
667-
# `terminationGracePeriodSeconds` to a value greater than or equal to the
668-
# configured `shutdown.grace-period`. Configure `shutdown.grace-period` in
669-
# `additionalConfigProperties` as `shutdown.grace-period=2m` (default is 2
670-
# minutes). Also configure `accessControl` because the `default` system
671-
# access control does not allow graceful shutdowns.
666+
# worker.lifecycle -- Worker container [lifecycle
667+
# events](https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/)
668+
#
669+
# Setting `worker.lifecycle` conflicts with `worker.gracefulShutdown`.
670+
#
672671
# @raw
673672
# Example:
674673
# ```yaml
675674
# preStop:
676675
# exec:
677-
# command: ["/bin/sh", "-c", "curl -v -X PUT -d '\"SHUTTING_DOWN\"' -H \"Content-type: application/json\" http://localhost:8081/v1/info/state"]
676+
# command: ["/bin/sh", "-c", "sleep 120"]
677+
# ```
678+
679+
gracefulShutdown:
680+
enabled: false
681+
gracePeriodSeconds: 120
682+
# worker.gracefulShutdown -- Configure [graceful
683+
# shutdown](https://trino.io/docs/current/admin/graceful-shutdown.html)
684+
# in order to ensure that workers terminate without affecting running queries,
685+
# given a sufficient grace period.
686+
# When enabled, the value of `worker.terminationGracePeriodSeconds` must be at least two times greater than the configured `gracePeriodSeconds`.
687+
# Enabling `worker.gracefulShutdown` conflicts with `worker.lifecycle`. When a custom
688+
# `worker.lifecycle` configuration needs to be used, graceful shutdown must be configured manually.
689+
#
690+
# @raw
691+
# Example:
692+
# ```yaml
693+
# gracefulShutdown:
694+
# enabled: true
695+
# gracePeriodSeconds: 120
678696
# ```
679697

680698
terminationGracePeriodSeconds: 30

‎test-graceful-shutdown-values.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
worker:
2+
gracefulShutdown:
3+
enabled: true
4+
gracePeriodSeconds: 60
5+
6+
terminationGracePeriodSeconds: 120

‎test.sh

+2-1
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ declare -A testCases=(
99
[overrides]="--set coordinatorNameOverride=coordinator-overridden,workerNameOverride=worker-overridden,nameOverride=overridden"
1010
[access_control_properties_values]="--values test-access-control-properties-values.yaml"
1111
[exchange_manager_values]="--values test-exchange-manager-values.yaml"
12+
[graceful_shutdown]="--values test-graceful-shutdown-values.yaml"
1213
)
1314

1415
function join_by {
@@ -23,7 +24,7 @@ NAMESPACE=trino-$(LC_ALL=C tr -dc 'a-z0-9' </dev/urandom | head -c 6 || true)
2324
HELM_EXTRA_SET_ARGS=
2425
CT_ARGS=(--charts=charts/trino --skip-clean-up --helm-extra-args="--timeout 2m")
2526
CLEANUP_NAMESPACE=true
26-
TEST_NAMES=(default single_node complete_values access_control_properties_values exchange_manager_values)
27+
TEST_NAMES=(default single_node complete_values access_control_properties_values exchange_manager_values graceful_shutdown)
2728

2829
usage() {
2930
cat <<EOF 1>&2

0 commit comments

Comments
 (0)