Now Kubernetes is a de facto standard solution for hosting cloud native applications but cluster hosting is expensive. It needs master nodes, worker nodes, networking, load balancers, etc. Recently I found a cheaper solution on Oracle cloud. 🙂

First of all, Kubernetes needs machines and memory. Oracle cloud gives us 2 AMD machines With 0.25 vCPU and 1GB ram, it is not enough for Kubernetes. We need to use a second compute option, ARM machines. We can use 4 vCPU and 24GB of memory and split it up to 4 virtual mechines. 2 vCPU and 12GB of ram is enough, even for Kubernetes master nodes. I started with two arm machines, 2CPU and 12GB memory each and kubespray for cluster configuration.

Kubespray is a powerfull and easy to use tool for configuration of kubernetes clusters, it can create clusters on AWS, and standalone machines. I don’t know Oracle cloud internals and I don’t need integration.

Cluster configuration

Setup Kubernetes cluster is very easy with kubespray, just create two machines, configure inventory and run ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml command. But in Oracle cloud there is a small pitfall. I use ubuntu machines, with a weird iptables config that blocks a lot of traffic.
I use script to disable problematic rules:

#!/bin/bash
# save iptables rules
sudo iptables-save > ~/iptables-rules
# remove all DROP and REJECT
grep -v "DROP" iptables-rules > tmpfile && mv tmpfile iptables-rules-mod
grep -v "REJECT" iptables-rules-mod > tmpfile && mv tmpfile iptables-rules-mod
# apply configuration
sudo iptables-restore < ~/iptables-rules-mod
# save iptables config
sudo netfilter-persistent save
sudo systemctl restart iptables

Kubespray should create a fully operational cluster on arm64 machines. I could stop here but I have another two free machines! Let’s create two more machines with an amd64 processor and add to cluster to get more compute power. 🧅

Multiarch cluster problem

I modified my inventory file and added two more machines as nodes. After that modification I ran kubespray again. Ansible finished without problem but nodes are NotReady! I checked and found a problem with the Calico. DaemonSet calico-node doesn’t work on new nodes because architecture mismatch. Looks like my calico configuration container image is not a multiarch image. 🙁 It is weird because official calico documentation uses different image, from dockerhub with multiarch support.

I didn’t know that a multiarch image existed when I configured my cluster. Therefore I decided to create two DaemonSets with calico and use nodeSelector to select the correct architecture. I just modified daemonSet that I downloaded from cluster using kubectl get daemonsets calico-node -n kube-system -o yaml > calico-arm64.yaml command. Code example for amd64, arm version uses arm64 arch and image tag.

...  
    - image: quay.io/calico/pod2daemon-flexvol:v3.19.2-amd64
...  
    nodeSelector:
        kubernetes.io/os: linux
        beta.kubernetes.io/arch: amd64
...

Multiarch deployments

Despite using two architectures I’d like to deploy my application on any node though two architectures. Nicely that docker can build multiarch images with buildx. I use github actions builder to build images. Example is in my repository with a simple ruby debug server: https://github.com/es1o/debug_server. Using multiarch images I can create only one deployment yaml. Kubernetes automatically chooses the correct architecture.

Conclusion

Kubernetes cluster doesn’t need all nodes with the same architecture if you take care about DaemonSets especially (e.g. kube-proxy, networking).