Kubernetes 3 node php

Read time: 15 minutes
Execution time: 30 minutes

Introduction

Getting started with Kubernetes can be quite hard. There are some very good books, like ‘Kubernetes in Action’ from Manning or ‘Kubernetes the hard way’ on Linux Academy. But you will certainly agree that learning works best when you get started with a nice project that is easy to understand or at least well documented.

The project I’m presenting here is a php page, served from 3 pods, spread over 3 nodes. An nginx ingress controller is used to enable ingress network traffic and cert-manager is used to create a certificate, signed by Let’s Encrypt. Here a high level overview:

Requirements

This guide is valid for nearly every cloud provided Kubernetes cluster. I clearly state ‘cloud provided’ for a reason. You can host your own Kubernetes cluster in Minikube, or have it self-hosted on physical or virtual machines. But using a cloud provided cluster, some extra needed resources, like load balancers or storage are provisioned automatically. This is very convenient. For my blog I’m using an Azure Kubernetes Service (AKS) cluster. But using another cloud will work as well with minimal changes.

You will also need an Ingress Controller. There are a lot of Ingress Controllers available like nginx, traefik, … In this guide, we are using nginx.

These days, we also want to protect our websites with a certificate. Therefore, we will also need cert-manager to request a certificate from Let’s Encrypt using the ACME protocol.

I will show you the commands to install the nginx Ingress Controller and cert-manager in a minute.

We also have an extra requirement, the nfs-server-provisioner. This will be explained in more detail in the storage chapter.

The last requirement we have is Helm, a package manager for Kubernetes. We will install it first, since we need it to install the rest of the requirements.

Installing requirements

Azure Kubernetes Service (AKS)

I won’t go too deep into detail for the installation of a Kubernetes cluster in Azure (AKS). This is a well documented process, which can be found here:

Kubernetes walkthrough

Helm

The installation of Helm is also documented well. The installation guide can be found here:

Installing Helm

You can install Helm on almost any operating system. Personally, I’m working with Windows 10, but I use the Windows Subsystem for Linux (WSL) with Ubuntu for managing AKS. I also like to recommend the new Windows Terminal, which supports tabs and multiple types of terminals (like PowerShell, PowerShell Core, CMD, Linux, …).

nginx Ingress controller

For the installation of the nginx Ingress Controller, we’ll use Helm that we have installed in the previous step:

# If you don't have the nginx-stable repo installed
$ helm repo add nginx-stable https://helm.nginx.com/stable
$ helm repo update

# When using helm 2
$ helm install nginx-stable/nginx-ingress --name website-ingress

# When using helm 3
$ helm install website-ingress nginx-stable/nginx-ingress

cert-manager

The installation of the cert-manager, that will handle the request of the certificate, is also done using helm and is therefore very straight-forward:

# If you don't have the cert-manager-stable repo installed
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update

# Assuming helm 3
$ helm install cert-manager jetstack/cert-manager

Storage

Since we will deploy a 3 node setup, we will need to use some kind of shared storage. There are a few ways to present storage to your pods, but the best way to do so is to make use of Persistent Volumes. This is an abstraction of the storage, which is very well explained in ‘Kubernetes in Action’, but in a nutshell explained:

An administrator creates a Persistent Volume (PV) using a Storage Class (SC) that is suited for the back-end storage. In AKS, the default Storage Class creates standard HDD managed disks. When the Persistent Volume is created, the developer, who doesn’t care about back-end disks, just needs to create a Persistent Volume Claim. The Persistent Volume Claim reserves a part of the Persistent Volume.

Another very important concept is the Access Mode. This can be ReadWriteOnce (RWO) or ReadWriteMany (RWX). The first one can only be accessed by one pod at a time while the latter can be accessed by multiple pods at once. In this example, using 3 nodes, we will need an RWX access policy.

In Azure you can choose for Managed Disks or Azure Files. Managed Disks only support RWO, where Azure Files also support RWX. At the time of writing this article, using Azure Files is very slow, even with Premium disks and for test purposes. Therefore, we have to work around this.

The workaround, to present Managed Disks to multiple pods, is to create an nfs-server-provisioner. This provisioner creates an NFS Storage Class, supporting RWX and connects to an existing Storage Class (e.g. default). With this intermediate Storage Controller, RWX can be provided to the pods.

To install the nfs-server-provisioner, helm is also used. But this time, some extra parameters are used to configure the back-end storage for the NFS Storage Class.

# When you don't have the stable repo installed
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com
$ helm repo update

$ helm install nfs stable/nfs-server-provisioner \
        --set persistence.enabled=true \
        --set persistence.size=20Gi \
        --set persistence.storageClass=default

Here’s the output of the storage classes with NFS when everything went well:

As you see, I don’t use kubectl, but k and I don’t write StorageClass, but sc. The main reason for this is because I’m as lazy as an IT Pro should be. And, playing around with Kubernetes makes you type kubectl so many times that it becomes simply too long… Therefore, I created an alias for the kubectl command. When specifying Kubernetes resource types, a short name can be used in a lot of cases. The list with all resource types with abbreviations can be found here.

Namespaces

I didn’t specify any namespace in the above commands, but Kubernetes uses namespaces to separate environments.

You can specify a namespace with both helm and kubectl by adding the –namespace parameter. When you are using the kubectl get command for requesting data, then you can specify the namespace as well, but you can also add the switch –all-namespaces, which will retrieve data from all namespaces.

Command examples

# Create a new namespace
$ kubectl create namespace <name>

# Request the pods in all namespaces
$ kubectl get pods --all-namespaces

# Set the default context of kubectl to a certain namespace
$ kubectl config set-context --namespace <name> --current

Important

Some resource types, like nodes, are not namespaced. This can also be found on the resource types page, linked above.

Deployment

Now, with all prerequisites in place, we can get started with the actual deployment. It consists of the following components:

• Replication Controller
• Service
• Persistent Volume Claim
• Cluster Issuer

Replication Controller

The Replication controller contains the definition of the pods with their containers. There is also a number of replicas specified. When deployed, the replication controller schedules a specified number of pods.

A volume is also created, pointing to the Persistent Volume Claim, and mounted to “/var/www/html”.

apiVersion: v1
kind: ReplicationController
metadata:
  name: website-rc
spec:
  replicas: 3
  selector:
    app: frontend
  template:
    metadata:
      name: website
      labels:
        app: frontend
    spec:
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: website-pvc
      containers:
      - name: webserver
        image: php:7.3-apache
        ports:
        - name: http 
          containerPort: 80
          protocol: TCP
        volumeMounts:
        - mountPath: "/var/www/html"
          name: data

Don’t deploy this deployment file yet as this is only a part of the deployment.

Service

There is no connection possible from outside, directly to a pod. To enable this, a service is needed. The service object creates an entry point, like in this case a cluster IP. The service will forward traffic to all the pods where the label matches the selector of the service.

apiVersion: v1
kind: Service
metadata:
  name: website
spec:
  selector:
    app: frontend
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

Persistent Volume Claim

The Persistent Volume Claim uses the NFS Storage Class. It is also necessary to define the ReadWriteMany (RWX) access mode to enable access from multiple pods at the same time. We’ll claim 2Gi (2 Gibibyte).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: website-pvc
spec:
  storageClassName: nfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

Ingress

The Ingress object handles external access to the service object. It points to the previously installed nginx Ingress Controller. In the specs, the host specification in the ingress controller, the hostname is specified to which the controller responds. If traffic hits the ingress controller, but pointing to a different host, then the traffic will be dropped. The path is the location within the url, after the hostname.

In the TLS section, the hostname is the name on the certificate. It obviously needs to match the host in the previous section. The secret contains the certificate and private key that is installed in the Ingress Controller.

There are also some annotations that create some settings:

• ingress.class: the ingress controller to use
• tls-acme: use ACME to create request a certificate
• cluster-issuer: the object that issues the certificate using cert-manager
• secure-backends: secure the incoming traffic

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: website-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: "website-prod"
    ingress.kubernetes.io/secure-backends: "true"
spec:
  rules:
  - host: kubernetes.groovesoundz.be
    http:
      paths:
      - path: /
        backend:
          serviceName: website
          servicePort: 80
  tls:
  - hosts:
    - kubernetes.groovesoundz.be
    secretName: website-prod

# Find the IP address of the ingress
$ kubectl get ingress

Important

Don’t forget to register the hostame in public DNS. It needs to be resolvable by your machine. This could be fixed with your hosts file. But Let’s Encrypt needs to be able to access the validation URL as well. And therefore, hosts file is not enough and public DNS should be registered.

ClusterIssuer

A Cluster Issuer is used to obtain certificates from a CA, like in this case Let’s Encrypt. The difference between an Issuer and a Cluster Issuer is that the latter can be referenced by objects in other namespaces.

The ACME server, specified in the definition, is obtained from Let’s Encrypt. A valid email address within the domain needs to be specified.

Since the cert-manager uses http01 as solver, it will create a url within the domain (e.g. http://kubernetes.groovesoundz.be/.well-known/acme-challenge/qlIfs-sflkjsdgflskdfFSDQ34Fffz3RZFDSS). This will prove the ownership of the domain. An ingress controller will point this path on the hostname to a pod from the cert-manager, which will approve the request of the certificate.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: website-prod
  namespace: cert-manager
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: christof@groovesoundz.be
    privateKeySecretRef:
      name: website-prod
    solvers:
    - http01:
       ingress:
         class: nginx

Deployment file

The deployment file consists of all the parts above with a line with 3 hyphens ‘—‘ (without the quotes obviously) in between.

Website.yaml

<ReplicaController>
---
<Service>
---
<PersistentVolumeClaim>
---
<Ingress>
---
<ClusterIssuer>

This deployment can also be found on my github: https://github.com/christofvg/kubernetes-3-node-php-site.

Deployment of the file

To deploy the actual deployment file, a deployment needs to be created in Kubernetes:

# Create a deployment, based on the deployment file

$ kubectl apply -f Website.yaml

Website content

The deployment creates the pods with shared storage. But this shared storage is empty of course. The point of this article is not to create a fancy website, but to show the workings of Kubernetes. Therefore, the content can consist of only one line that displays the pod’s name (hostname).

Here’s how to add the content:

# find out the names of the pods
$ kubectl get pods

# Open a bash shell on one of the pods
$ kubectl exec -it <name of one of the pods> bash

# If not already, ls to '/var/www/html'
$ ls /var/www/html

# Add the content to index.php
$ echo '<?php echo gethostname();?>' > index.php

From now on, the hostname (the name of the pod) will be shown when the pod is accessed through the ingress and service. The load will be balanced over the pods, so when you refresh, it changes. It might stay the same a few times. This is because of the way a browser handles connections. When you use curl to connect to the URL, then it changes every time since every time a new connection is started.

Conclusion

We have created a high available web server environment, complete with load balancing and certificate. I also tried to do this in a way, explaining things I found hard at first.

Please use the comments below to share your experiences.