Kubernetes is awesome!



I've been using AWS/ECS/EC2 with Docker Cloud on and off for a while now. Whilst I was impressed with the flexibility and potential, I found the AWS platform to be difficult to use, it has a poor UX and I find docker cloud to be unreliable and buggy.

I'd heard of Kubernetes and Google Cloud before but hadn't found time to check it out. I was a little bored last weekend so thought I'd at least do the hello world, or Hello Node in this case.

I was awestruck at how quickly I was set-up and running. It didn't run into endless errors and misconfigurations either, I followed the tutorial, it made sense, and it worked first time. I had a node app running in the cloud.

Google also gave me £250 worth of free credits to play with for my first month. Bonus!

The UI is much sleeker and much easier to reason about than AWS's. Kubernetes is also declarative in style, your cluster configurations, and changes can be tracked within your repository.

I'm going to share a bit of what I've learnt so far. I'll just quickly go through some of the concepts and terminology. Which for me, was probably the most difficult part of the process.

Deployments

A deployment is a description of your final desired state. You're telling Google Cloud that you just want 2 of x containers running, on y and z ports, and you don't care how it does that, as long as it's done, and maintained in that state until told otherwise.

Here's an example of a deployment file:

apiVersion: extensions/v1beta1  
kind: Deployment  
metadata:  
  name: backend
spec:  
  replicas: 2
  template:
    metadata:
      labels:
        app: app-service
        tier: backend
    spec:
      containers:
      - name: service-a-container
        image: gcr.io/applied-area-732/service-a:latest
        ports:
        - containerPort: 8080
        imagePullPolicy: Always
      - name: service-b-container
        image: gcr.io/applied-area-732/service-b:latest
        ports:
        - containerPort: 7000
        imagePullPolicy: Always

apiVersion: extensions/v1beta1 is the sdk/api version to use. This seems to be the go to for now, despite being ominously named v1beta1.

kind: Deployment - this means this action is a 'Deployment' type. Deployments are a replacement for 'Replication Controllers'.

Deployments let you describe your clouds desired state. You can create separate deployment files for separate groups etc.

spec: - Spec is a list of basic configuration options, such as the amount of replicated nodes you wish to run. You will notice labels, also. This allows you to defined groups of interconnected, or related 'pods'. For example you could have a 'payments' label and 'backend' as your 'tier'. You can also use your 'app' name to define which pod to connect to which service.

The next line you'll see another spec, this is our list of containers, including the images to use, the ports to run on etc.

You will also notice that I'm deploying two services within this pod.

Services

A service is a gateway to one or more pods. Think of a service as a set of IP tables rules which route incoming traffic to the correct pods. This is done through matching up port numbers, and using the selectors field in the configuration.

Services also perform some high-level cluster management and service discovery under the hood. A service sits in front of your pods, giving them access to the real world. There are different types of services, the most notable one is LoadBalancer which automatically sets up a load balancer for your pods. Which is pretty cool!

Here's an example of a service:

apiVersion: v1  
kind: Service  
metadata:  
  name: backend
  labels:
    app: app-service
    tier: backend
spec:  
  type: LoadBalancer
  ports:
  - port: 8080
    name: service-a-port
    targetPort: 8080
  - port: 7000
    name: service-b-port
    targetPort: 7000
  selector:
    app: app-service
    tier: backend

On the first line we define which version of the Kubernetes API to use (be aware these don't match the deployment API version numbers). Then we're setting this config as a service config type. Then we're including our labels, which allow us to define which group of pods and services this service should give access to.

Then onto spec which allows us to define multiple ports to open up publicly, note that in order to use multiple ports, you must include a port name for each. Then finally, we include a selector which let's you define which pods should have access to this service. As long as your selector names and ports match your pod names and ports. This should now be public facing.

Replica Sets

In our deployment example, you'll notice replicas: 2. Replica sets are what were previously known as Replication Controllers. Replica sets are a definition of how many of each service should be present in a cluster. These are defined by your deployment.

Pods

A pod is one or more containers which depend upon each other. For example, if you have an ecommerce API, split into microservices. You may have an inventory service, this could have it's own queue, it's own database, and it's own REST API. These would exist as a pod, so that they are provisioned together as one group. These pods can then be scaled horizontally.

Gluing it all together

Typically, as there's a lot of command line wizardry going on, I'll go through each of the commands you'll likely need to use.

Pushing your image to GCR

I'm using Google's container repository, as it's bundled together nicely with Google's Cloud services. It's very easy to use, and it you're using Google Cloud, it's nice to have everything in the same space.

$ gcloud docker push gcr.io/<PROJECT_NAME>/<REPO_NAME>:<TAG>

This will push your built docker image to GCR. Please be aware that the tag should be unique in order for changes to be picked up by deployments. What I do is use the short git hash ($ git rev-parse --short HEAD) to form the Docker tag for my images, but you could use your Jenkins build number, or anything unique.

Create a cluster

gcloud container clusters create <YOUR_CLUSTER_NAME> --machine-type g1-small

Create a deployment

To actually take a deployment file and action those events, you would use the following command:
$ kubectl create -f ./deployment.yml

Or to update a deployment with changes:

$ kubectl replace -f ./deployment.yml

Create a service

$ kubectl create -f ./my-service.yml

To update a service (note we use apply here instead of replace):

$ kubectl apply -f ./updated-service.yml

Full example

Here's a full example of a deploy script, and an update script.

#!/bin/bash
set -e

# Set working directory as ./scripts
parent_path=$( cd "$(dirname "${BASH_SOURCE}")" pwd -P )  
cd "$parent_path"

hash=$(git rev-parse --short HEAD)

echo "Building services"

# Build services
docker build -t gcr.io/applied-area-732/service-a:$hash service-a

docker build -t gcr.io/applied-area-732/service-b:$hash service-b

echo "Creating Cluster"

# Create cluster
gcloud container clusters create bandzest --machine-type g1-small

echo "Pushing containers to cloud"

# Push services to cloud
gcloud docker push gcr.io/applied-area-732/service-a:latest

gcloud docker push gcr.io/applied-area-732/service-b:latest

echo "Creating Deployment"

# Nodejs script to generate kubectl file from templates, to include the git hash.
./scripts/generate-deployment.js

# Deploy Replication Controller
kubectl create -f deploy.yml

echo "Creating Service"

# Expose to the internet
kubectl create -f service.yml  

Now the Nodejs script for creating our templates:

#!/usr/bin/env node

require('shelljs/global')

const fs = require('fs')

/**
 * Get git short hash
 *
 * @param {String}
 * @param {Object}
 * @param {Function}
 */
exec('git rev-parse --short HEAD', { silent: true }, (code, stdout) => {

  // deploy template
  const deployTemplate = './scripts/deploy.tmpl.yml'
  const deployUITemplate = './scripts/deploy-frontend.tmpl.yml'

  // deploy ouput
  const outputFile = './scripts/deploy.yml'
  const outputUIFile = './scripts/deploy-frontend.yml'

  // Open the deploy template
  fs.readFile(deployTemplate, (err, contents) => {
    if (err) console.log(err)

    // Replace GIT_HASH var with actual git has
    const newContents = contents.toString()
      .replace('{{ GIT_HASH }}', stdout.trim())
      .replace('{{ GIT_HASH }}', stdout.trim())

    // Write file to output
    fs.writeFile(outputFile, newContents, (err) => {
      if (err) console.log(err)
      console.log(`Wrote deployment file ${outputFile}`)
    })
  })
})

Now our update script:

#!/bin/bash
set -e

# Set working directory as ./scripts
parent_path=$( cd "$(dirname "${BASH_SOURCE}")" pwd -P )  
cd "$parent_path"

hash=$(git rev-parse --short HEAD)

echo "Building services"

docker build -t gcr.io/applied-area-732/service-a:$hash service-a

docker build -t gcr.io/applied-area-732/service-b:$hash service-b

echo "Updating services"

gcloud docker push gcr.io/applied-area-732/service-a:$hash

gcloud docker push gcr.io/applied-area-732/service-b:$hash

echo "Push updated services"

./scripts/generate-deployment.js

kubectl replace -f ./scripts/deploy.yml

kubectl apply -f ./scripts/service.yml  

I hope this is a useful overview of Kubernetes!

Full example repo here