Experimenting with flux2 on microk8s
In which I try to get flux2 working on microk8s...
Flux is a tool for configuring Kubernetes clusters using Git repositories. Changes in configuration pushed to a GitHub repo are then pushed to the cluster. This enables a GitOps-style approach to application infrastructure management.
microk8s is a minimal but complete Kubernetes distribution available from Canonical. It is fully containerized as a snap package which is easy to install and run.
Spun up a dedicated VPS on Vultr, SSH'd in, and ran OS updates:
apt update ; apt upgrade -y
snap install microk8s --classic
Confirmed microk8s was up and running:
microk8s status --wait-ready
The snap of flux is flux 1.x, so to get install flux2, it is necessary to use the install script:
curl -s https://toolkit.fluxcd.io/install.sh | sudo bash
Ran the flux prerequisites check:
flux check --pre
flux2 doesn't detect the kubectl alias.
Looked at the source of flux2 where error is being generated, it is looking for kubectl in the OS PATH variable.
Create a kubectl wrapper script:
Copy and paste the following wrapper script:
#!/bin/bash microk8s kubectl "$@"
Made the wrapper executable:
chmod u+x /usr/local/bin/kubectl
Re-ran flux prerequisites check:
flux check --pre
kubectl is detected, but kubernetes client initialization is now failing.
- Looked at the source of flux2 where error is being generated.
- Used kubectl config view to get the IP of our cluster and set the KUBERNETES_MASTER variable, did not work.
- Checked ~/.kube for a config file, not there. Found the kubeconfig used by microk8s deep in the snap, linked it to ~/.kube/config, success!
Set my GitHub PAT and username:
export GITHUB_TOKEN=<token> export GITHUB_USER=sirredbeard
Bootstrapped my repository:
flux bootstrap github \ --owner=$GITHUB_USER \ --repository=flux-sandbox \ --branch=main \ --path=./clusters/my-cluster \ --personal
unable to clone 'ssh://firstname.lastname@example.org/sirredbeard/flux-sandbox', error: dial tcp: lookup github.com: Try again
Confirmed GitHub repository was created:
Re-ran bootstrap command, no change.
Looked for logs. Probably in source-controller:
kubectl -n flux-system logs source-controller-65f5b6cb7c-ct7k8
Looks like source-controller can't reach notification-controller or github.com, odd.
Confirm DNS is enabled:
microk8s enable dns
There is some networking thing here I am not seeing. Taking a break to return to it later...
Continuing: I ran the issue by the Kubernetes team, something about it just did not seem right, and it turns out this is a DNS bug.
The temporary workaround stated is to convert to a non-HA cluster, which I did as followed:
microk8s disable ha-cluster microk8s disable dns microk8s stop microk8s start microk8s enable dns
Re-ran the bootstrap script, and success!
I then connected to the VPS over SSH using VS Code and git cloned my flux-sandbox repo to my home folder:
I then added the repository for a sample app, podinfo, from the flux2 Get Started guide to my deployment:
flux create source git podinfo \ --url=https://github.com/stefanprodan/podinfo \ --branch=master \ --interval=30s \ --export > ./clusters/my-cluster/podinfo-source.yaml
This created the postinfo-source.yaml manifest (below), which I then committed and pushed to GitHub from VS Code:
Next, I then deployed the podinfo app by creating a Flux Kustomization manifest for podinfo:
flux create kustomization podinfo \ --source=podinfo \ --path="./kustomize" \ --prune=true \ --validation=client \ --interval=5m \ --export > ./clusters/my-cluster/podinfo-kustomization.yaml
Which I then committed and pushed.
And confirmed running:
kubectl get pods --all-namespaces
Pausing for now.