Experimenting with flux2 on microk8s

Flux is a tool for configuring Kubernetes clusters using Git repositories. Changes in configuration pushed to a GitHub repo are then pushed to the cluster. This enables a GitOps-style approach to application infrastructure management.

microk8s is a minimal but complete Kubernetes distribution available from Canonical. It is fully containerized as a snap package which is easy to install and run.

Spun up a dedicated VPS on Vultr, SSH'd in, and ran OS updates:

apt update ; apt upgrade -y

Installed microk8s:

snap install microk8s --classic

Confirmed microk8s was up and running:

microk8s status --wait-ready

The snap of flux is flux 1.x, so to get install flux2, it is necessary to use the install script:

Installed flux2:

curl -s https://toolkit.fluxcd.io/install.sh | sudo bash

Ran the flux prerequisites check:

flux check --pre

flux2 doesn't detect the kubectl alias.

Looked at the source of flux2 where error is being generated, it is looking for kubectl in the OS PATH variable.

Create a kubectl wrapper script:

nano /usr/local/bin/kubectl

Copy and paste the following wrapper script:

#!/bin/bash
microk8s kubectl "$@"

Made the wrapper executable:

chmod u+x /usr/local/bin/kubectl

Tested executable:

kubectl version

Re-ran flux prerequisites check:

flux check --pre

kubectl is detected, but kubernetes client initialization is now failing.

Troubleshooting attempted:

• Looked at the source of flux2 where error is being generated.
• Used kubectl config view to get the IP of our cluster and set the KUBERNETES_MASTER variable, did not work.

• Checked ~/.kube for a config file, not there. Found the kubeconfig used by microk8s deep in the snap, linked it to ~/.kube/config, success!

Set my GitHub PAT and username:

export GITHUB_TOKEN=<token>
export GITHUB_USER=sirredbeard

Bootstrapped my repository:

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-sandbox \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal

Received error:

unable to clone 'ssh://git@github.com/sirredbeard/flux-sandbox', error: dial tcp: lookup github.com: Try again

Confirmed GitHub repository was created:

Re-ran bootstrap command, no change.

Looked for logs. Probably in source-controller:

kubectl -n flux-system logs source-controller-65f5b6cb7c-ct7k8

Looks like source-controller can't reach notification-controller or github.com, odd.

Confirm DNS is enabled:

microk8s enable dns

There is some networking thing here I am not seeing. Taking a break to return to it later...

Continuing: I ran the issue by the Kubernetes team, something about it just did not seem right, and it turns out this is a DNS bug.

The temporary workaround stated is to convert to a non-HA cluster, which I did as followed:

microk8s disable ha-cluster
microk8s disable dns
microk8s stop
microk8s start
microk8s enable dns

Re-ran the bootstrap script, and success!

I then connected to the VPS over SSH using VS Code and git cloned my flux-sandbox repo to my home folder:

I then added the repository for a sample app, podinfo, from the flux2 Get Started guide to my deployment:

flux create source git podinfo \
  --url=https://github.com/stefanprodan/podinfo \
  --branch=master \
  --interval=30s \
  --export > ./clusters/my-cluster/podinfo-source.yaml

This created the postinfo-source.yaml manifest (below), which I then committed and pushed to GitHub from VS Code:

Next, I then deployed the podinfo app by creating a Flux Kustomization manifest for podinfo:

flux create kustomization podinfo \
  --source=podinfo \
  --path="./kustomize" \
  --prune=true \
  --validation=client \
  --interval=5m \
  --export > ./clusters/my-cluster/podinfo-kustomization.yaml

Which I then committed and pushed.

And confirmed running:

kubectl get pods --all-namespaces

Pausing for now.