Skip to content

Commit 6e4ea16

Browse files
tormath1tormath1
authored
Merge pull request #22 from flatcar/tormath1/brigthbox
brightbox: kubernetes bootstrap example
2 parents 7000b58 + 3f136fc commit 6e4ea16

9 files changed

+366
-0
lines changed

‎brightbox/README.md

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Flatcar Provisioning Automation for Brightbox
2+
3+
This repository provides tools to automate Kubernetes provisioning on [Brightbox][brightbox] using [Terraform][terraform] and Flatcar via the Systemd sysext approach: https://www.flatcar.org/docs/latest/container-runtimes/getting-started-with-kubernetes/#deploy-a-kubernetes-cluster-with-flatcar
4+
5+
:warning: This is really for demo purposes but it can serve as a foundation (for example do not pass the admin configuration through HTTP for workers to join) :warning:
6+
7+
## Features
8+
9+
- Minimal configuration required (demo deployment works with default settings w/o any customisation, just run `terraform apply`!).
10+
- Deploy one or multiple workers.
11+
12+
## Prerequisites
13+
14+
1. Brightbox credentials: `api_client`, `api_secret`.
15+
2. A public SSH key to install on the control plane
16+
17+
## HowTo
18+
19+
This will create a server in 'gb1-a' using a medium instance size for the control plane and small instance sizes for the three workers.
20+
See "Customisation" below for advanced settings.
21+
22+
1. Clone the repo.
23+
2. Add credentials and a SSH key in a `terraform.tfvars` file, expected credentials name can be found in `provider.tf`
24+
3. Run
25+
```shell
26+
terraform init
27+
```
28+
4. Plan and apply.
29+
Invoke Terraform:
30+
```shell
31+
terraform plan
32+
terraform apply
33+
```
34+
35+
Terraform will print the control plane information (ipv4) after deployment concluded. You can now easily fetch the kubernetes `admin` configuration via a secure channel:
36+
37+
```
38+
$ scp core@<IP from the output>:/home/core/.kube/config ~/.kube/config
39+
$ kubectl get nodes
40+
NAME STATUS ROLES AGE VERSION
41+
srv-cruzw.gb1.brightbox.com NotReady <none> 55s v1.29.2
42+
srv-fltor.gb1.brightbox.com NotReady control-plane 72s v1.29.2
43+
srv-gvzhx.gb1.brightbox.com NotReady <none> 59s v1.29.2
44+
srv-mipnf.gb1.brightbox.com NotReady <none> 60s v1.29.2
45+
```
46+
47+
From now, you can operate the Kubernetes cluster as usual (deploy CNI, deploy workloads, etc.)
48+
49+
_NOTE_:
50+
* Server IP address can be found at any moment after deployment by running `terraform output`
51+
* If you update server configuration(s) in `server-configs` and re-run `terraform apply`, the instance will be **replaced**.
52+
Consider adding [`create_before_destroy`](https://www.terraform.io/docs/configuration/meta-arguments/lifecycle.html#syntax-and-arguments) to the `brightbox_server` resource in [`compute.tf`](compute.tf) to avoid services becoming unavailable during reprovisioning.
53+
54+
### Customisation
55+
56+
The provisioning automation can be customised via settings in `terraform.tfvars`:
57+
- `ssh_keys`: SSH public keys to add to core user's `authorized_keys` (needed for fetching the Kubernetes configuration)
58+
- `release_channel`: Select one of "lts", "stable", "beta", or "alpha".
59+
Read more about channels [here](https://www.flatcar.org/releases).
60+
- `flatcar_version`: Select the desired Flatcar version for the given channel (default to "current", which is the latest).
61+
- `zone`: Where to deploy servers
62+
- `control_plane_type`: Which instance type used for deploying the controle plane
63+
- `worker_type`: Which instance type used for deploying the workers
64+
- `kubernetes_version`: The Kubernetes version to deploy (NOTE: It has to be released on the Flatcar sysext bakery: https://github.com/flatcar/sysext-bakery/releases/tag/latest)
65+
- `workers`: How many workers to deploy
66+
67+
[butane]: https://www.flatcar.org/docs/latest/provisioning/config-transpiler/configuration/
68+
[brightbox]: https://www.brightbox.com/
69+
[terraform]: https://www.terraform.io/

‎brightbox/compute.tf

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
data "brightbox_image" "flatcar" {
2+
name = "^flatcar-${var.release_channel}.*server$"
3+
arch = "x86_64"
4+
official = true
5+
most_recent = true
6+
}
7+
8+
resource "brightbox_server" "control-plane" {
9+
image = data.brightbox_image.flatcar.id
10+
name = "control-plane"
11+
zone = var.zone
12+
type = var.control_plane_type
13+
user_data = data.ct_config.config-control-plane.rendered
14+
server_groups = [brightbox_server_group.kubernetes.id]
15+
depends_on = [brightbox_firewall_policy.kubernetes]
16+
}
17+
18+
resource "brightbox_server" "worker" {
19+
count = var.workers
20+
image = data.brightbox_image.flatcar.id
21+
name = "worker-${count.index}"
22+
zone = var.zone
23+
type = var.worker_type
24+
user_data = data.ct_config.config-worker.rendered
25+
}
26+
27+
data "ct_config" "config-control-plane" {
28+
strict = true
29+
content = templatefile("${path.module}/server-configs/control-plane.yaml.tmpl", {
30+
kubernetes_version = var.kubernetes_version
31+
})
32+
snippets = [
33+
data.template_file.core_user.rendered
34+
]
35+
}
36+
37+
data "ct_config" "config-worker" {
38+
strict = true
39+
content = templatefile("${path.module}/server-configs/worker.yaml.tmpl", {
40+
kubernetes_version = var.kubernetes_version
41+
control_plane_ip = brightbox_cloudip.control-plane.public_ipv4
42+
})
43+
}
44+
45+
data "template_file" "core_user" {
46+
template = file("${path.module}/core-user.yaml.tmpl")
47+
vars = {
48+
ssh_keys = jsonencode(var.ssh_keys)
49+
}
50+
}
51+
52+
resource "brightbox_cloudip" "control-plane" {
53+
target = brightbox_server.control-plane.interface
54+
name = "control-plane public address"
55+
}

‎brightbox/core-user.yaml.tmpl

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
variant: flatcar
2+
version: 1.0.0
3+
4+
passwd:
5+
users:
6+
- name: core
7+
ssh_authorized_keys: ${ssh_keys}

‎brightbox/network.tf

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
resource "brightbox_server_group" "kubernetes" {
2+
name = "Kubernetes server group"
3+
}
4+
5+
resource "brightbox_firewall_policy" "kubernetes" {
6+
name = "Kubernetes firewall policy"
7+
server_group = brightbox_server_group.kubernetes.id
8+
}
9+
10+
resource "brightbox_firewall_rule" "kubernetes_api" {
11+
destination_port = 6443
12+
protocol = "tcp"
13+
source = "any"
14+
description = "Kubernetes API access from anywhere"
15+
firewall_policy = brightbox_firewall_policy.kubernetes.id
16+
}
17+
18+
resource "brightbox_firewall_rule" "ssh" {
19+
destination_port = 22
20+
protocol = "tcp"
21+
source = "any"
22+
description = "SSH access from anywhere"
23+
firewall_policy = brightbox_firewall_policy.kubernetes.id
24+
}
25+
26+
resource "brightbox_firewall_rule" "workers" {
27+
protocol = "tcp"
28+
source = data.brightbox_server_group.default.id
29+
firewall_policy = brightbox_firewall_policy.kubernetes.id
30+
}
31+
32+
resource "brightbox_firewall_rule" "internet" {
33+
protocol = "tcp"
34+
destination = "any"
35+
firewall_policy = brightbox_firewall_policy.kubernetes.id
36+
}
37+
38+
data "brightbox_server_group" "default" {
39+
name = "^default$"
40+
}

‎brightbox/outputs.tf

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
output "ipv4-control-plane" {
2+
value = brightbox_cloudip.control-plane.public_ipv4
3+
}

‎brightbox/provider.tf

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
terraform {
2+
required_version = ">= 0.14.0"
3+
required_providers {
4+
brightbox = {
5+
source = "brightbox/brightbox"
6+
version = "3.4.3"
7+
}
8+
ct = {
9+
source = "poseidon/ct"
10+
version = "0.11.0"
11+
}
12+
template = {
13+
source = "hashicorp/template"
14+
version = "~> 2.2.0"
15+
}
16+
}
17+
}
18+
19+
provider "brightbox" {
20+
apiclient = var.api_client
21+
apisecret = var.api_secret
22+
}
23+

‎brightbox/server-configs/control-plane.yaml.tmpl

+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
version: 1.0.0
3+
variant: flatcar
4+
storage:
5+
links:
6+
- target: /opt/extensions/kubernetes/kubernetes-${kubernetes_version}-x86-64.raw
7+
path: /etc/extensions/kubernetes.raw
8+
hard: false
9+
files:
10+
- path: /etc/sysupdate.kubernetes.d/kubernetes.conf
11+
contents:
12+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes.conf
13+
- path: /etc/sysupdate.d/noop.conf
14+
contents:
15+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
16+
- path: /opt/extensions/kubernetes/kubernetes-${kubernetes_version}-x86-64.raw
17+
contents:
18+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${kubernetes_version}-x86-64.raw
19+
systemd:
20+
units:
21+
- name: systemd-sysupdate.timer
22+
enabled: true
23+
- name: systemd-sysupdate.service
24+
dropins:
25+
- name: kubernetes.conf
26+
contents: |
27+
[Service]
28+
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
29+
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
30+
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
31+
ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
32+
- name: kubeadm.service
33+
enabled: true
34+
contents: |
35+
[Unit]
36+
Description=Kubeadm service
37+
Requires=containerd.service
38+
After=containerd.service coreos-metadata.service
39+
Requires=coreos-metadata.service
40+
ConditionPathExists=!/etc/kubernetes/kubelet.conf
41+
[Service]
42+
EnvironmentFile=/run/metadata/flatcar
43+
ExecStartPre=/usr/bin/kubeadm init --control-plane-endpoint "$${COREOS_OPENSTACK_IPV4_PUBLIC}:6443"
44+
ExecStartPre=/usr/bin/mkdir -p /home/core/.kube /var/www
45+
ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /home/core/.kube/config
46+
ExecStartPre=/usr/bin/cp /etc/kubernetes/admin.conf /var/www
47+
ExecStartPre=/usr/bin/chmod a+r /var/www/admin.conf
48+
ExecStart=/usr/bin/chown -R core:core /home/core/.kube
49+
[Install]
50+
WantedBy=multi-user.target
51+
- name: nginx.service
52+
enabled: true
53+
contents: |
54+
[Unit]
55+
Description=NGINX
56+
After=docker.service
57+
Requires=docker.service
58+
[Service]
59+
TimeoutStartSec=0
60+
ExecStartPre=-/usr/bin/docker rm --force nginx1
61+
ExecStart=/usr/bin/docker run --name nginx1 -p 8080:80 --volume "/var/www:/usr/share/nginx/html:ro" --pull always --log-driver=journald docker.io/nginx:1
62+
ExecStop=/usr/bin/docker stop nginx1
63+
Restart=always
64+
RestartSec=5s
65+
[Install]
66+
WantedBy=multi-user.target

‎brightbox/server-configs/worker.yaml.tmpl

+47
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
version: 1.0.0
3+
variant: flatcar
4+
storage:
5+
links:
6+
- target: /opt/extensions/kubernetes/kubernetes-${kubernetes_version}-x86-64.raw
7+
path: /etc/extensions/kubernetes.raw
8+
hard: false
9+
files:
10+
- path: /etc/sysupdate.kubernetes.d/kubernetes.conf
11+
contents:
12+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes.conf
13+
- path: /etc/sysupdate.d/noop.conf
14+
contents:
15+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
16+
- path: /opt/extensions/kubernetes/kubernetes-${kubernetes_version}-x86-64.raw
17+
contents:
18+
source: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${kubernetes_version}-x86-64.raw
19+
systemd:
20+
units:
21+
- name: systemd-sysupdate.timer
22+
enabled: true
23+
- name: systemd-sysupdate.service
24+
dropins:
25+
- name: kubernetes.conf
26+
contents: |
27+
[Service]
28+
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
29+
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
30+
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
31+
ExecStartPost=/usr/bin/sh -c "[[ $(cat /tmp/kubernetes) != $(cat /tmp/kubernetes-new) ]] && touch /run/reboot-required"
32+
- name: kubeadm.service
33+
enabled: true
34+
contents: |
35+
[Unit]
36+
Description=Kubeadm service
37+
Requires=containerd.service
38+
After=containerd.service
39+
ConditionPathExists=!/etc/kubernetes/kubelet.conf
40+
[Service]
41+
Restart=on-failure
42+
StartLimitInterval=0
43+
RestartSec=10
44+
ExecStartPre=/usr/bin/curl -fsSL http://${control_plane_ip}:8080/admin.conf -o /tmp/admin.conf
45+
ExecStart=/usr/bin/kubeadm join --discovery-file /tmp/admin.conf
46+
[Install]
47+
WantedBy=multi-user.target

‎brightbox/variables.tf

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
variable "ssh_keys" {
2+
type = list(string)
3+
default = []
4+
description = "Additional SSH public keys for user 'core'."
5+
}
6+
7+
variable "release_channel" {
8+
type = string
9+
description = "Release channel"
10+
default = "stable"
11+
12+
validation {
13+
condition = contains(["lts", "stable", "beta", "alpha"], var.release_channel)
14+
error_message = "release_channel must be lts, stable, beta, or alpha."
15+
}
16+
}
17+
18+
variable "api_client" {
19+
type = string
20+
description = "Brightbox API client"
21+
}
22+
23+
variable "api_secret" {
24+
type = string
25+
description = "Brightbox API secret"
26+
}
27+
28+
variable "zone" {
29+
type = string
30+
description = "Brightbox zone"
31+
default = "gb1-a"
32+
}
33+
34+
variable "control_plane_type" {
35+
type = string
36+
description = "Brightbox control plane instance type"
37+
default = "4gb.ssd"
38+
}
39+
40+
variable "worker_type" {
41+
type = string
42+
description = "Brightbox worker instance type"
43+
default = "1gb.ssd"
44+
}
45+
46+
variable "kubernetes_version" {
47+
type = string
48+
description = "Kubernetes version"
49+
default = "v1.29.2"
50+
}
51+
52+
variable "workers" {
53+
type = number
54+
description = "Number of workers"
55+
default = "3"
56+
}

0 commit comments

Comments
 (0)