GitOps – declarative Continuous Deployment on Kubernetes in simple steps

Intro

In this article I’m going to show you very basic and simple example which describes GitOps concept – new pattern in DevOps world which helps to keep infrastructure stable and more recoverable. By using git as operations tool we can see the whole history of changes and we are able to say who and when changed something. As example I want to show you really simple Continuous Deployment pipeline of docker images on Kubernetes cluster.

Main assumptions are:

  • Git repository is single source of truth about state of the system
  • If state change in repository it will be automatically synchronized

Proposed architecture:

Article - gitops cd

Deployment design

Let’s say that we have small company with 3-4 projects managed by single DevOps team on the same infrastructure and Kubernetes cluster. Every project is based on microservices architecture with 4-5 services. Every project should have at least 2-3 environments:

  • Development – with CD pipeline from development branch of every microservice
  • Staging/Production – stable environment where microservices are deployed only on-demand when management says that version is stable, well tested and ready to handle clients. In our example from implementation point of view staging and production are the same.

Docker images

To keep simple mapping between the source code and artifact which will be deployed on some environment, docker images should be tagged using git commit hash – that’s ID which allows us to simply identify which code is deployed so in case of debugging some complex environments that’s really helpful. We can achieve that easily in CI pipeline which produce docker images by using script like:

HEAD=$(git rev-parse HEAD)

docker build -t registry.gitlab.com/myproject/microservice:${HEAD} .

docker push registry.gitlab.com/myproject/microservice:${HEAD}

Environments state repository

Heart of GitOps concept is state repository which should be source of truth for whole system. It means that in case of failure we should be able to recover the system using configuration stored in git repository. In our example we will use simpler concept – store only git hash of particular microservice what will represent version of application deployed on particular environment. It should allow us to make our infrastructure really reproducible but not on 100% as deployment configuration like environment variables can change behavior of the system. It means that in real GitOps pattern we should also store in such repository configuration of deployment like kubernetes yaml files, infrastructure descriptors like terraform and ansible files etc. It’s better but much more complicated as changes pushed to repository should be automatically synchronized by applying the diff.

Repository structure

Screenshot from 2018-12-23 12-26-08

So every deployment has own directory where name of directory is also name of the namespace in Kubernetes cluster. Every microservice in particular deployment has one file where git commit hash is stored. There is also one special file images_path which contains docker image path. Docker image name with tag will be constructed using following pattern:

content of images_path file (e.g. registry.gitlab.com/mycompany/project/development) + deployment file name (e.g. authentication) + “:” + content of deployment file (e.g. 36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2)

Example: registry.gitlab.com/mycompany/project/development/authentication:36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2

There is also one special directory sync-agent which will be described in next paragraph.

Sync agent

To avoid making manual updates on environment it’s required to have some synchronization agent which will read state of repository and apply changes in deployment. To achieve that we will use simple CronJob on Kubernetes which will run periodically and use kubectl to update image of particular deployment. Sync agent will be created per Kubernetes namespace to improve isolation between environments. So at first we must create proper RBAC permissions to allow our job to update deployments.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sync-agent
  namespace: project1-development

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sync-agent
  namespace: project1-development
rules:
- apiGroups: ["extensions"]
  resources: ["deployments"]
  verbs: ["get","list","patch","update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sync-agent
  namespace: project1-development
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sync-agent
subjects:
- kind: ServiceAccount
  name: sync-agent
  namespace: project1-development

In above configuration we create service account for sync-agent and give to it permissions to manipulate deployments within his namespace.

Next important thing is to create secret with deploy key – such deploy key should be configured to have read-only access to state repository so sync agent can clone the repo and read deployment files.

apiVersion: v1
kind: Secret
metadata:
  name: sync-agent-deploy-key
  namespace: project1-development
type: Opaque
data:
  id_rsa: 

Final part is to create sync-agent CronJob which will run every minute and make synchronization:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: sync-agent
  namespace: project1-development
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 30
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sync-agent
          containers:
          - name: sync-agent
            image: lachlanevenson/k8s-kubectl:v1.10.3
            command:
            - /bin/sh
            - -c
            args:
            - apk update &&
              apk add git &&
              apk add openssh &&
              mkdir -p /root/.ssh && cp /root/key/id_rsa /root/.ssh/id_rsa &&
              chmod 0600 /root/.ssh/id_rsa &&
              ssh-keyscan -t rsa gitlab.com >> /root/.ssh/known_hosts &&
              mkdir -p /deployment && cd /deployment &&
              git clone git@gitlab.com:my-project/environments-state.git &&
              cd environments-state/sync-agent &&
              NAMESPACE=project1-development ./sync-repository-and-kubernetes-state.sh
            volumeMounts:
            - name: sync-agent-deploy-key
              mountPath: /root/key/
          volumes:
          - name: sync-aginfrastructure ent-deploy-key
            secret:
              secretName: sync-agent-deploy-key
          restartPolicy: Never

Args and command section are a little hacked as we just use public image with kubectl we must install additional things at beginning of the container to keep example simple – probably in real usage we should create docker image with pre-installed tools and push it to some private registry.

Comments to lines:
7 – run job every minute
22-24 – install required tools
25-26 – ssh private key is mounted as file but we must copy it to proper place and set proper chmod to be able to use it
27 – add gitlab.com to known hosts file so git will be able to clone repo
29-31 – clone repository and cd to sync-agent dir where script sync-repository-and-kubernetes-state.sh is located

Code of the sync script:

#!/usr/bin/env sh
set -e
set -x

if [ -z ${NAMESPACE+x} ]; then echo "NAMESPACE env var is empty! Cannot proceed"; exit 1; fi

cd ../$NAMESPACE

IMAGES_PATH=$(cat images_path)
for deployment_file in *.deployment; do
    DEPLOYMENT_HASH=$(cat $deployment_file)
    DEPLOYMENT_NAME=$(echo $deployment_file | sed 's/\.deployment//')
    kubectl set image --namespace $NAMESPACE deployment/$DEPLOYMENT_NAME $DEPLOYMENT_NAME=${IMAGES_PATH}/${DEPLOYMENT_NAME}:${DEPLOYMENT_HASH}
done

Code is really simple – script takes NAMESPACE as parameter and then iterate over all deployment files making kubectl set image on every deployment. Kubernetes will do nothing when you try to set image with the same value which is already in deployment so making such set many times is idempotent. When some new image hash appears kubectl set image cause rolling upgrade on deployment.

Deployment procedure

Whole infrastructure is in place so now to create CD pipeline from development branch we must just place code which will:

  • read current git HEAD hash
  • tag docker with git HEAD hash and push it
  • commit to state repository new hash to file and push it

After such actions we can expect in around 1 minute that sync-agent will clone new state and make synchronization on Kubernetes cluster.

To make on-demand deployment on some stable environment we can just clone repository on our PC and place proper hashes in files manually and again sync-agent will do the work for us.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s