Skip to content

Commit aaf9bf7

Browse files
mburke5678mburke5678
committed
Kubelet-Controller Manager communication Profiles (WorkerLatencyProfiles)
1 parent 239f034 commit aaf9bf7

File tree

6 files changed

+311
-0
lines changed

6 files changed

+311
-0
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2030,6 +2030,9 @@ Topics:
20302030
- Name: Enabling features using FeatureGates
20312031
File: nodes-cluster-enabling-features
20322032
Distros: openshift-enterprise,openshift-origin
2033+
- Name: Improving cluster stability in high latency environments using worker latency profiles
2034+
File: nodes-cluster-worker-latency-profiles
2035+
Distros: openshift-enterprise,openshift-origin
20332036
- Name: Remote worker nodes on the network edge
20342037
Dir: edge
20352038
Distros: openshift-enterprise

‎modules/nodes-cluster-worker-latency-profiles-about.adoc

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * nodes/clusters/nodes-cluster-worker-latency-profiles
4+
// * nodes/edge/nodes-edge-remote-workers. ??
5+
// * post_installation_configuration/cluster-tasks ??
6+
7+
:_content-type: CONCEPT
8+
[id="nodes-cluster-worker-latency-profiles-about_{context}"]
9+
= Understanding worker latency profiles
10+
11+
Worker latency profiles are multiple sets of carefully-tuned values for the `node-status-update-frequency`, `node-monitor-grace-period`, `default-not-ready-toleration-seconds` and `default-unreachable-toleration-seconds` parameters. These parameters let you control the reaction of the cluster to latency issues without needing to determine the best values manually.
12+
13+
All worker latency profiles configure the following parameters:
14+
15+
--
16+
* `node-status-update-frequency`. Specifies the amount of time in seconds that a kubelet updates its status to the Kubernetes Controller Manager Operator.
17+
* `node-monitor-grace-period`. Specifies the amount of time in seconds that the Kubernetes Controller Manager Operator waits for an update from a kubelet before marking the node unhealthy and adding the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint to the node.
18+
* `default-not-ready-toleration-seconds`. Specifies the amount of time in seconds after marking a node unhealthy that the Kubernetes Controller Manager Operator waits before evicting pods from that node.
19+
* `default-unreachable-toleration-seconds`. Specifies the amount of time in seconds after marking a node unreachable that the Kubernetes Controller Manager Operator waits before evicting pods from that node.
20+
--
21+
22+
[IMPORTANT]
23+
====
24+
Manually modifying the `node-monitor-grace-period` parameter is not supported.
25+
====
26+
27+
While the default configuration works in most cases, {product-title} offers two other worker latency profiles for situations where the network is experiencing higher latency than usual. The three worker latency profiles are described in the following sections:
28+
29+
Default worker latency profile:: With the `Default` profile, each kubelet reports its node status to the Kubelet Controller Manager Operator (kube controller) every 10 seconds. The Kubelet Controller Manager Operator checks the kubelet for a status every 5 seconds.
30+
+
31+
The Kubernetes Controller Manager Operator waits 40 seconds for a status update before considering that node unhealthy. It marks the node with the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint and evicts the pods on that node. If a pod on that node has the `NoExecute` toleration, the pod gets evicted in 300 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter.
32+
+
33+
[cols="2,1,2,1"]
34+
|===
35+
| Profile | Component | Parameter | Value
36+
37+
.4+| Default
38+
| kubelet
39+
| `node-status-update-frequency`
40+
| 10s
41+
42+
| Kubelet Controller Manager
43+
| `node-monitor-grace-period`
44+
| 40s
45+
46+
| Kubernetes API Server
47+
| `default-not-ready-toleration-seconds`
48+
| 300s
49+
50+
| Kubernetes API Server
51+
| `default-unreachable-toleration-seconds`
52+
| 300s
53+
54+
|===
55+
56+
Medium worker latency profile:: Use the `MediumUpdateAverageReaction` profile if the network latency is slightly higher than usual.
57+
+
58+
The `MediumUpdateAverageReaction` profile reduces the frequency of kubelet updates to 20 seconds and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 2 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter.
59+
+
60+
The Kubernetes Controller Manager Operator waits for 2 minutes to consider a node unhealthy. In another minute, the eviction process starts.
61+
+
62+
[cols="2,1,2,1"]
63+
|===
64+
| Profile | Component | Parameter | Value
65+
66+
.4+| MediumUpdateAverageReaction
67+
| kubelet
68+
| `node-status-update-frequency`
69+
| 20s
70+
71+
| Kubelet Controller Manager
72+
| `node-monitor-grace-period`
73+
| 2m
74+
75+
| Kubernetes API Server
76+
| `default-not-ready-toleration-seconds`
77+
| 60s
78+
79+
| Kubernetes API Server
80+
| `default-unreachable-toleration-seconds`
81+
| 60s
82+
83+
|===
84+
85+
Low worker latency profile:: Use the `LowUpdateSlowReaction` profile if the network latency is extremely high.
86+
+
87+
The `LowUpdateSlowReaction` profile reduces the frequency of kubelet updates to 1 minute and changes the period that the Kubernetes Controller Manager Operator waits for those updates to 5 minutes. The pod eviction period for a pod on that node is reduced to 60 seconds. If the pod has the `tolerationSeconds` parameter, the eviction waits for the period specified by that parameter.
88+
+
89+
The Kubernetes Controller Manager Operator waits for 5 minutes to consider a node unhealthy. In another minute, the eviction process starts.
90+
+
91+
[cols="2,1,2,1"]
92+
|===
93+
| Profile | Component | Parameter | Value
94+
95+
.4+| LowUpdateSlowReaction
96+
| kubelet
97+
| `node-status-update-frequency`
98+
| 1m
99+
100+
| Kubelet Controller Manager
101+
| `node-monitor-grace-period`
102+
| 5m
103+
104+
| Kubernetes API Server
105+
| `default-not-ready-toleration-seconds`
106+
| 60s
107+
108+
| Kubernetes API Server
109+
| `default-unreachable-toleration-seconds`
110+
| 60s
111+
112+
|===
113+

‎modules/nodes-cluster-worker-latency-profiles-using.adoc

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * nodes/clusters/nodes-cluster-worker-latency-profiles
4+
// * Need to determine if these are good locations:
5+
// * nodes/edge/nodes-edge-remote-workers
6+
// * post_installation_configuration/cluster-tasks
7+
8+
:_content-type: PROCEDURE
9+
[id="nodes-cluster-worker-latency-profiles-using_{context}"]
10+
= Using worker latency profiles
11+
12+
To implement a worker latency profile to deal with network latency, edit the `node.config` object to add the name of the profile. You can change the profile at any time as latency increases or decreases.
13+
14+
You must move one worker latency profile at a time. For example, you cannot move directly from the `Default` profile to the `LowUpdateSlowReaction` worker latency profile. You must move from the `default` worker latency profile to the `MediumUpdateAverageReaction` profile first, then to `LowUpdateSlowReaction`. Similarly, when returning to the default profile, you must move from the low profile to the medium profile first, then to the default.
15+
16+
[NOTE]
17+
====
18+
You can also configure worker latency profiles upon installing an {product-title} cluster.
19+
====
20+
21+
.Procedure
22+
23+
To move from the default worker latency profile:
24+
25+
. Move to the medium worker latency profile:
26+
27+
.. Edit the `node.config` object:
28+
+
29+
[source,terminal]
30+
----
31+
$ oc edit nodes.config/cluster
32+
----
33+
34+
.. Add `spec.workerLatencyProfile: MediumUpdateAverageReaction`:
35+
+
36+
.Example `node.config` object
37+
[source,yaml]
38+
----
39+
apiVersion: config.openshift.io/v1
40+
kind: Node
41+
metadata:
42+
annotations:
43+
include.release.openshift.io/ibm-cloud-managed: "true"
44+
include.release.openshift.io/self-managed-high-availability: "true"
45+
include.release.openshift.io/single-node-developer: "true"
46+
release.openshift.io/create-only: "true"
47+
creationTimestamp: "2022-07-08T16:02:51Z"
48+
generation: 1
49+
name: cluster
50+
ownerReferences:
51+
- apiVersion: config.openshift.io/v1
52+
kind: ClusterVersion
53+
name: version
54+
uid: 36282574-bf9f-409e-a6cd-3032939293eb
55+
resourceVersion: "1865"
56+
uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
57+
spec:
58+
workerLatencyProfile: MediumUpdateAverageReaction <1>
59+
60+
...
61+
----
62+
<1> Specifies the medium worker latency policy.
63+
+
64+
Scheduling on each worker node is disabled as the change is being applied.
65+
+
66+
When all nodes return to the `Ready` condition, you can use the following command to look in the Kubernetes Controller Manager to ensure it was applied:
67+
+
68+
[source,terminal]
69+
----
70+
$ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5
71+
----
72+
+
73+
.Example output
74+
[source,terminal]
75+
----
76+
...
77+
- lastTransitionTime: "2022-07-11T19:47:10Z"
78+
reason: ProfileUpdated
79+
status: "False"
80+
type: WorkerLatencyProfileProgressing
81+
- lastTransitionTime: "2022-07-11T19:47:10Z" <1>
82+
message: all static pod revision(s) have updated latency profile
83+
reason: ProfileUpdated
84+
status: "True"
85+
type: WorkerLatencyProfileComplete
86+
- lastTransitionTime: "2022-07-11T19:20:11Z"
87+
reason: AsExpected
88+
status: "False"
89+
type: WorkerLatencyProfileDegraded
90+
- lastTransitionTime: "2022-07-11T19:20:36Z"
91+
status: "False"
92+
...
93+
----
94+
<1> Specifies that the profile is applied and active.
95+
96+
. Optional: Move to the low worker latency profile:
97+
98+
.. Edit the `node.config` object:
99+
+
100+
[source,terminal]
101+
----
102+
$ oc edit nodes.config/cluster
103+
----
104+
105+
.. Change the `spec.workerLatencyProfile` value to `LowUpdateSlowReaction`:
106+
+
107+
.Example `node.config` object
108+
[source,yaml]
109+
----
110+
apiVersion: config.openshift.io/v1
111+
kind: Node
112+
metadata:
113+
annotations:
114+
include.release.openshift.io/ibm-cloud-managed: "true"
115+
include.release.openshift.io/self-managed-high-availability: "true"
116+
include.release.openshift.io/single-node-developer: "true"
117+
release.openshift.io/create-only: "true"
118+
creationTimestamp: "2022-07-08T16:02:51Z"
119+
generation: 1
120+
name: cluster
121+
ownerReferences:
122+
- apiVersion: config.openshift.io/v1
123+
kind: ClusterVersion
124+
name: version
125+
uid: 36282574-bf9f-409e-a6cd-3032939293eb
126+
resourceVersion: "1865"
127+
uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
128+
spec:
129+
workerLatencyProfile: LowUpdateSlowReaction <1>
130+
131+
...
132+
----
133+
<1> Specifies to use the low worker latency policy.
134+
+
135+
Scheduling on each worker node is disabled as the change is being applied.
136+
137+
To change the low profile to medium or change the medium to low, edit the `node.config` object and set the `spec.workerLatencyProfile` parameter to the appropriate value.
138+

‎nodes/clusters/nodes-cluster-worker-latency-profiles.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
:_content-type: ASSEMBLY
2+
:context: nodes-cluster-worker-latency-profiles
3+
[id="nodes-cluster-worker-latency-profiles"]
4+
= Improving cluster stability in high latency environments using worker latency profiles
5+
include::_attributes/common-attributes.adoc[]
6+
7+
toc::[]
8+
9+
10+
11+
// The following include statements pull in the module files that comprise
12+
// the assembly. Include any combination of concept, procedure, or reference
13+
// modules required to cover the user story. You can also include other
14+
// assemblies.
15+
16+
17+
include::snippets/worker-latency-profile-intro.adoc[]
18+
19+
These worker latency profiles are three sets of parameters that are pre-defined with carefully tuned values that control the reaction of the cluster to latency issues without needing to determine the best values manually.
20+
21+
You can configure worker latency profiles when installing a cluster or at any time you notice increased latency in your cluster network.
22+
23+
include::modules/nodes-cluster-worker-latency-profiles-about.adoc[leveloffset=+1]
24+
25+
include::modules/nodes-cluster-worker-latency-profiles-using.adoc[leveloffset=+1]
26+

‎post_installation_configuration/cluster-tasks.adoc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -503,6 +503,15 @@ include::modules/machineset-delete-policy.adoc[leveloffset=+2]
503503

504504
include::modules/nodes-scheduler-node-selectors-cluster.adoc[leveloffset=+2]
505505

506+
[id="post-worker-latency-profiles"]
507+
== Improving cluster stability in high latency environments using worker latency profiles
508+
509+
include::snippets/worker-latency-profile-intro.adoc[]
510+
511+
include::modules/nodes-cluster-worker-latency-profiles-about.adoc[leveloffset=+2]
512+
513+
include::modules/nodes-cluster-worker-latency-profiles-using.adoc[leveloffset=+2]
514+
506515
[id="post-install-creating-infrastructure-machinesets-production"]
507516
== Creating infrastructure machine sets for production environments
508517

‎snippets/worker-latency-profile-intro.adoc

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Text snippet included in the following modules:
2+
//
3+
// * nodes/clusters/nodes-cluster-worker-latency-profiles
4+
// * nodes/edge/nodes-edge-remote-workers
5+
// * post_installation_configuration/cluster-tasks
6+
7+
8+
:_content-type: SNIPPET
9+
10+
All nodes send heartbeats to the Kubernetes Controller Manager Operator (kube controller) in the {product-title} cluster every 10 seconds, by default. If the cluster does not receive heartbeats from a node, {product-title} responds using several default mechanisms.
11+
12+
For example, if the Kubernetes Controller Manager Operator loses contact with a node after a configured period:
13+
14+
. The node controller on the control plane updates the node health to `Unhealthy` and marks the node `Ready` condition as `Unknown`.
15+
16+
. In response, the scheduler stops scheduling pods to that node.
17+
18+
. The on-premise node controller adds a `node.kubernetes.io/unreachable` taint with a `NoExecute` effect to the node and schedules any pods on the node for eviction after five minutes, by default.
19+
20+
This behavior can cause problems if your network is prone to latency issues, especially if you have nodes at the network edge. In some cases, the Kubernetes Controller Manager Operator might not receive an update from a healthy node due to network latency. The Kubernetes Controller Manager Operator would then evict pods from the node even though the node is healthy. To avoid this problem, you can use _worker latency profiles_ to adjust the frequency that the kubelet and the Kubernetes Controller Manager Operator wait for status updates before taking action. These adjustments help to ensure that your cluster runs properly in the event that network latency between the control plane and the worker nodes is not optimal.
21+
22+
These worker latency profiles are three sets of parameters that are pre-defined with carefully tuned values that let you control the reaction of the cluster to latency issues without needing to determine the best values manually.

0 commit comments

Comments
 (0)