GitOps – declarative Continuous Deployment on Kubernetes in simple steps

Intro

In this article I’m going to show you very basic and simple example which describes GitOps concept – new pattern in DevOps world which helps to keep infrastructure stable and more recoverable. By using git as operations tool we can see the whole history of changes and we are able to say who and when changed something. As example I want to show you really simple Continuous Deployment pipeline of docker images on Kubernetes cluster.

Main assumptions are:

  • Git repository is single source of truth about state of the system
  • If state change in repository it will be automatically synchronized

Proposed architecture:

Article - gitops cd

Deployment design

Let’s say that we have small company with 3-4 projects managed by single DevOps team on the same infrastructure and Kubernetes cluster. Every project is based on microservices architecture with 4-5 services. Every project should have at least 2-3 environments:

  • Development – with CD pipeline from development branch of every microservice
  • Staging/Production – stable environment where microservices are deployed only on-demand when management says that version is stable, well tested and ready to handle clients. In our example from implementation point of view staging and production are the same.

Docker images

To keep simple mapping between the source code and artifact which will be deployed on some environment, docker images should be tagged using git commit hash – that’s ID which allows us to simply identify which code is deployed so in case of debugging some complex environments that’s really helpful. We can achieve that easily in CI pipeline which produce docker images by using script like:

HEAD=$(git rev-parse HEAD)

docker build -t registry.gitlab.com/myproject/microservice:${HEAD} .

docker push registry.gitlab.com/myproject/microservice:${HEAD}

Environments state repository

Heart of GitOps concept is state repository which should be source of truth for whole system. It means that in case of failure we should be able to recover the system using configuration stored in git repository. In our example we will use simpler concept – store only git hash of particular microservice what will represent version of application deployed on particular environment. It should allow us to make our infrastructure really reproducible but not on 100% as deployment configuration like environment variables can change behavior of the system. It means that in real GitOps pattern we should also store in such repository configuration of deployment like kubernetes yaml files, infrastructure descriptors like terraform and ansible files etc. It’s better but much more complicated as changes pushed to repository should be automatically synchronized by applying the diff.

Repository structure

Screenshot from 2018-12-23 12-26-08

So every deployment has own directory where name of directory is also name of the namespace in Kubernetes cluster. Every microservice in particular deployment has one file where git commit hash is stored. There is also one special file images_path which contains docker image path. Docker image name with tag will be constructed using following pattern:

content of images_path file (e.g. registry.gitlab.com/mycompany/project/development) + deployment file name (e.g. authentication) + “:” + content of deployment file (e.g. 36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2)

Example: registry.gitlab.com/mycompany/project/development/authentication:36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2

There is also one special directory sync-agent which will be described in next paragraph.

Sync agent

To avoid making manual updates on environment it’s required to have some synchronization agent which will read state of repository and apply changes in deployment. To achieve that we will use simple CronJob on Kubernetes which will run periodically and use kubectl to update image of particular deployment. Sync agent will be created per Kubernetes namespace to improve isolation between environments. So at first we must create proper RBAC permissions to allow our job to update deployments.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sync-agent
  namespace: project1-development

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sync-agent
  namespace: project1-development
rules:
- apiGroups: ["extensions"]
  resources: ["deployments"]
  verbs: ["get","list","patch","update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sync-agent
  namespace: project1-development
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sync-agent
subjects:
- kind: ServiceAccount
  name: sync-agent
  namespace: project1-development

In above configuration we create service account for sync-agent and give to it permissions to manipulate deployments within his namespace.

Next important thing is to create secret with deploy key – such deploy key should be configured to have read-only access to state repository so sync agent can clone the repo and read deployment files.

apiVersion: v1
kind: Secret
metadata:
  name: sync-agent-deploy-key
  namespace: project1-development
type: Opaque
data:
  id_rsa: 

Final part is to create sync-agent CronJob which will run every minute and make synchronization:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: sync-agent
  namespace: project1-development
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 30
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sync-agent
          containers:
          - name: sync-agent
            image: lachlanevenson/k8s-kubectl:v1.10.3
            command:
            - /bin/sh
            - -c
            args:
            - apk update &&
              apk add git &&
              apk add openssh &&
              mkdir -p /root/.ssh && cp /root/key/id_rsa /root/.ssh/id_rsa &&
              chmod 0600 /root/.ssh/id_rsa &&
              ssh-keyscan -t rsa gitlab.com >> /root/.ssh/known_hosts &&
              mkdir -p /deployment && cd /deployment &&
              git clone git@gitlab.com:my-project/environments-state.git &&
              cd environments-state/sync-agent &&
              NAMESPACE=project1-development ./sync-repository-and-kubernetes-state.sh
            volumeMounts:
            - name: sync-agent-deploy-key
              mountPath: /root/key/
          volumes:
          - name: sync-aginfrastructure ent-deploy-key
            secret:
              secretName: sync-agent-deploy-key
          restartPolicy: Never

Args and command section are a little hacked as we just use public image with kubectl we must install additional things at beginning of the container to keep example simple – probably in real usage we should create docker image with pre-installed tools and push it to some private registry.

Comments to lines:
7 – run job every minute
22-24 – install required tools
25-26 – ssh private key is mounted as file but we must copy it to proper place and set proper chmod to be able to use it
27 – add gitlab.com to known hosts file so git will be able to clone repo
29-31 – clone repository and cd to sync-agent dir where script sync-repository-and-kubernetes-state.sh is located

Code of the sync script:

#!/usr/bin/env sh
set -e
set -x

if [ -z ${NAMESPACE+x} ]; then echo "NAMESPACE env var is empty! Cannot proceed"; exit 1; fi

cd ../$NAMESPACE

IMAGES_PATH=$(cat images_path)
for deployment_file in *.deployment; do
    DEPLOYMENT_HASH=$(cat $deployment_file)
    DEPLOYMENT_NAME=$(echo $deployment_file | sed 's/\.deployment//')
    kubectl set image --namespace $NAMESPACE deployment/$DEPLOYMENT_NAME $DEPLOYMENT_NAME=${IMAGES_PATH}/${DEPLOYMENT_NAME}:${DEPLOYMENT_HASH}
done

Code is really simple – script takes NAMESPACE as parameter and then iterate over all deployment files making kubectl set image on every deployment. Kubernetes will do nothing when you try to set image with the same value which is already in deployment so making such set many times is idempotent. When some new image hash appears kubectl set image cause rolling upgrade on deployment.

Deployment procedure

Whole infrastructure is in place so now to create CD pipeline from development branch we must just place code which will:

  • read current git HEAD hash
  • tag docker with git HEAD hash and push it
  • commit to state repository new hash to file and push it

After such actions we can expect in around 1 minute that sync-agent will clone new state and make synchronization on Kubernetes cluster.

To make on-demand deployment on some stable environment we can just clone repository on our PC and place proper hashes in files manually and again sync-agent will do the work for us.

 

Useful Docker #1 – Seed container at runtime

Docker is one of my favorite tool in IT – it gives me really fast access to already configured&installed tools. Using that feature I can cover many cases using simple docker container and some magic around. In this article series I’m going to show you how to use docker in creative way and how to debug problems inside docker containers.

Today something about CMD, magic in runtime and docker commit.

I have assumption that you have basic knowledge about docker and some experience in using that great tool – if not I suggest to check that site: https://docs.docker.com/engine/docker-overview/

My environment in this article:

  • Docker version 17.12.0-ce, build c97c6d6
  • Ubuntu 17.10
  • zsh 5.2 (x86_64-ubuntu-linux-gnu)
  • tmux 2.5

Everything should also work in docker on windows (this one using hyper-v) – if there is problem somewhere please contact me on my linkedin

g-gif-update

Seed your container in runtime

Dockerhub provides many images (as mentioned here in 2016 was 400k) with useful defaults but these defaults rarely meet our requirements. To deal with that situation we can take many strategies but one of my favorite is overriding default docker command with some magic.

Let’s say that we want to host jakubbujny.com blog in docker container and switch my site logo to

57761

just for fun!

Required steps:

  • Start container
  • Download jakubbujny.com site content recursive
  • Start web hosting
  • Change image

We need to start with some static web content hosting like https://hub.docker.com/_/nginx/

We have command from this site (with port publish):

docker run -p 8080:80 --name some-nginx -v /some/content:/usr/share/nginx/html:ro -d nginx

Ok but after staring this container we can see only welcome site from nginx – how to inject content there? There are 2 ways:

  • Add content to docker image – useful if we want to use that image many times
  • Override run command and download content before web server start – our approach because we are fast&furious

So what’s default command in nginx? To check that we should go to origin Dockerfile like this and at end we see:

CMD ["nginx", "-g", "daemon off;"]

Cool, let’s try this way:

docker run -it -p 8080:80 nginx bash -c "cd /usr/share/nginx/html && wget --recursive --no-parent --no-check-certificate https://jakubbujny.com ; nginx -g 'daemon off;'"

What’s happening there:

  • docker – 😉
  • run – start new container
  • -it – interactive (get stdout, pass stdin)
  • -p 8080:80 – publish internal nginx 80 port onto our machine 8080 so we can access nginx using localhost:8080
  • bash -c ” ” – that’s little magic to pass some long command with &&, ; – bash man says:
    -c string If the -c option is present,  then  commands  are  read  from
                 string.   If  there  are arguments after the string, they are
                 assigned to the positional parameters, starting with $0.
  • cd… – sure
  • wget – we need to download recursive my site
  • nginx -g – start nginx (use default command from Dockerfile)

Ok so let’s run it… woooops!!! Something is not working ;(

bash: wget: command not found

Please remember that every docker container is isolated virtual OS  (so actually it’s not but you can think about docker in this way 😉 ) – it means that every docker image has different tools installed. Sadly nginx image doesn’t contain wget tool but okey dokey – we can install wget using apt-get inside container, no worries:

docker run -it -p 8080:80 nginx bash -c "apt-get update && apt-get install --no-install-recommends --no-install-suggests -y wget && cd /usr/share/nginx/html && wget --recursive --no-parent --no-check-certificate https://jakubbujny.com ; nginx -g 'daemon off;'"

What’s happening there:

  • sudo not needed because inside container we are root
  • apt-get update – very important! Please remember to run that command before any apt-get install inside containers to get valid result
  • apt-get install – here -y is important because that command will run in non-interactive mode what means that we need auto-confirm that we are sure to install following package

So after running this command everything seems fine – wget is downloading my site and nginx is starting but at localhost:8080 I see:

Screenshot from 2018-02-18 12-14-04

Damn that’s not cool….

Debug

To debug that situation we need to get into container and see whats happening in file system after wget command. To do that we need to start with:

docker ps

And output:

Screenshot from 2018-02-18 12-16-45

The most important there is Container Id – we’re going to use that ID to get into container, open new console and try (change container id to yours because it’s random string):

docker exec -it 69cb502ab0d1 bash

After that command we are in new shell process inside container in interactive mode. Let’s check what’s under /usr/share/nginx/html:

drwxr-xr-x 1 root root 4.0K Feb 18 11:20 . 
drwxr-xr-x 1 root root 4.0K Dec 26 18:16 .. 
-rw-r--r-- 1 root root 537 Dec 26 11:11 50x.html 
-rw-r--r-- 1 root root 612 Dec 26 11:11 index.html 
drwxr-xr-x 7 root root 4.0K Feb 18 11:20 jakubbujny.com

Ok so wget downloaded my site into jakubbujny.com directory – try in web browser:

http://localhost:8080/jakubbujny.com/

You should see my blog 🙂

Change image

Stay please in our shell process inside container. Changing site logo is so easy, just type: vim /usr/share/nginx/html/jakubbujny.com/index.html omg there is no vim

tenor

<disclaimer> Many popular images don’t have basic tools like vim or even bash because every additional tool not needed at runtime means bigger image size to download for users </disclaimer>

apt-get install vim

I believe that you are VIM WIZARD LVL 300 but if not just type:

ESC /site-logo ENTER

To find my logo section and change next <img> tag src to (should help: ESC a ) : https://jakubbujny.files.wordpress.com/2018/02/57761.png

Then:

ESC :wq ENTER

Go to web browser and hit F5:

Screenshot from 2018-02-18 12-36-26.png

That was so easy, was it?

Commit container into image

Last step is to convert container into image.

PLEASE DO NOT USE THAT IN PRODUCTION – FOLLOW INFRASTRUCTURE AS CODE PATTERN

It means that we can say to docker: hey! I have here some container where I installed some tools and modified some state, please convert that container into image so I can reuse that manual configuration many times

Command:

docker commit 69cb502ab0d1 my-awesome-image:1.0

So again we are using Container ID and docker is creating for us image with name my-awesome-image:1.0 – let’s check if that works:

docker run -it -p 8081:80 my-awesome-image:1.0 bash -c "nginx -g 'daemon off;'"

What’s happening:

  • Run container from committed image
  • Override command again – docker set my-awesome-image’s cmd to that one with wget but we don’t need that here! We committed whole container so jakubbujny.com site is already in image – if we don’t need latest version we can override again cmd to start only web server

Go under:

http://localhost:8081/jakubbujny.com/

And you should see my blog and docker logo instead mine.

Conclusion

As you can see docker is quite powerful – when using with care you can really do magic cases. I hope that was helpful for you – I decided to divide docker article into series because I have many more examples how to use that tool in creative way. See you in next article!