Provision microservice’s pipeline on Jenkins using CustomResourceDefinition and Operator on Kubernetes

1_l_MvKlG3R7VCC6aGt53w_A

Overview

In this article I want to show you how to create custom resource in Kubernetes so we can create just another resource which provision CI/CD pipeline on Jenkins for microservice. To achieve such goal we will use operator-sdk CLI and write code in Go to implement integration with Jenkins using API.

Everything will happen in local environment on Minikube – Jenkins will be also deployed there using official Helm chart. Repositories will be created on Github and Dockerhub. Github respositories related to that article:

https://github.com/jakubbujny/article-jenkins-pipeline-crd

https://github.com/jakubbujny/article-microservice1

https://github.com/jakubbujny/article-microservice2

https://github.com/jakubbujny/article-jenkins-pipeline-crd

Jenkins deployment on Kubernetes

Jenkins will be deployed on Kubernetes using official Helm chart which contains already Kubernetes plugin to spawn build agents as separated PODs.

At first we should start Minikube with memory increased as Jenkins and agents are Java processes so they are consuming quite big amount of memory:

minikube start –memory=4096

after start we can deploy Jenkins using following script:

#!/usr/bin/env bash

helm init --wait

helm install \
 --name jenkins stable/jenkins \
 --set master.csrf.defaultCrumbIssuer.enabled=false \
 --set master.tag=2.194 \
 --set master.serviceType=ClusterIP \
 --set master.installPlugins[0]="kubernetes:1.18.1" --set master.installPlugins[1]="workflow-aggregator:2.6" --set master.installPlugins[2]="credentials-binding:1.19" --set master.installPlugins[3]="git:3.11.0" --set master.installPlugins[4]="workflow-job:2.33" \
 --set master.installPlugins[5]="job-dsl:1.76"

Lines:

  • 7 – disabling CSRF make that example simpler as we don’t must deal with issuing/sending crumbs in API requests
  • 9 – we must change Jenkins service type as by default it starts as LoadBalancer what won’t work on Minikube
  • 10 – those are default plugins in Helm chart, provided in such strange form because of some issue in Helm – more info https://stackoverflow.com/questions/48316330/how-to-set-multiple-values-with-helm
  • 11 – We need job dsl plugin what will be described in next section

after those operations Jenkins should start – to access it we can use following command:

kubectl port-forward svc/jenkins 8080:8080

kubectl will create proxy for us so we can see Jenkins UI under localhost:8080. Login is admin and password can be obtained by using following command:

printf $(kubectl get secret –namespace default jenkins -o jsonpath=”{.data.jenkins-admin-password}” | base64 –decode);echo

Seed job

Main concept will be based on seed job – Jenkins job DSL script which will provide pipelines for microservices. Such seed job will contain following code:

projects.split(',').each { project ->
  pipelineJob(project) {
    definition {
      cpsScm {
        scm {
          git {
            remote {
              url("https://github.com/jakubbujny/article-${project}.git")
            }
            branch("*/master")
          }
        }
        triggers {
           scm("* * * * *")
       }
        lightweight()
        scriptPath('Jenkinsfile.groovy')
      }
    }
  }
}

Projects variable will be provided as job parameter – that parameter will be modified by CRD Operator on Kubernetes when new resource is created.

Job DSL script for each project (microservice) will create pipeline job with Github project as source. That pipeline will use Jenkinsfile located in microservice’s repository and will have trigger based on repository pull so every minute repository will be pulled but pipeline will be triggered only when changes are detected.

Jenkinsfile.groovy source

microserviceName = "microservice1"

pipeline {
    agent {
        kubernetes {
            //cloud 'kubernetes'
            label 'mypod'
            yaml """
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: cicd
  containers:
  - name: docker
    image: docker:1.11
    command: ['cat']
    tty: true
    volumeMounts:
    - name: dockersock
      mountPath: /var/run/docker.sock
  - name: kubectl
    image: ubuntu:18.04
    command: ['cat']
    tty: true
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock
"""
        }
    }
    stages {
        stage('Build Docker image') {
            steps {
                checkout scm
                container('docker') {
                    script {
                        def image = docker.build("digitalrasta/article-${microserviceName}:${BUILD_NUMBER}")
                        docker.withRegistry( '', "dockerhub") {
                            image.push()
                        }
                    }
                }
            }
        }
        stage('Deploy') {
            steps {
                container('kubectl') {
                    script {
                        sh "apt-get update && apt-get install -y curl"
                        sh "curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl"
                        sh "chmod +x ./kubectl"
                        sh "mv ./kubectl /usr/local/bin/kubectl"
                        def checkDeployment = sh(script: "kubectl get deployments | grep ${microserviceName}", returnStatus: true)
                        if(checkDeployment != 0) {
                            sh "kubectl apply -f deploy/deploy.yaml"
                        }
                        sh "kubectl set image deployment/${microserviceName} ${microserviceName}=digitalrasta/article-${microserviceName}:${BUILD_NUMBER}"
                    }
                }
            }
        }
    }
}

That pipeline spawn POD on Kubernetes which will contain 3 containers – but we see definition only of 2 of them as 3rd container will be JNLP Jenkins agent. First container is used to build docker image with our microservice, tag it with BUILD_NUMBER, push to Dockerhub and second is Ubuntu where kubectl is installed to make deployment.

CD is simply made by updating docker image in particular deployment so it will be automatically pulled by Kubernetes from Dockerhub.

Please be also aware about ServiceAccount which must be installed on cluster named “cicd”:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cicd

---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cicd
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["jakubbujny.com"]
    resources: ["jenkinspipelines"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cicd
subjects:
  - kind: ServiceAccount
    name: cicd
roleRef:
  kind: Role
  name: cicd
  apiGroup: rbac.authorization.k8s.io

Cicd ServiceAccount has permissions to manipulate deployments and also to make operations over some special API jakubbujny.com and jenkinspipelines – that’s our CRD which will be described in next section. That permission is not really needed as “jenkinspipeline” resource should be installed by cluster admin but I left that to make example more clear.

CustomResourceDefinition and Operator

The final part is to make our own Operator with API definition. To do that we need Github repository and operator-sdk so we can start with following command

operator-sdk new jenkins-pipeline-operator –repo github.com/jakubbujny/jenkins-pipeline-operator

that command will create basic folders structure with boilerplate code.

As next we want to add our own API and Controller code which will react on changes in that API:

operator-sdk add api –api-version=jakubbujny.com/v1alpha1 –kind=JenkinsPipeline

operator-sdk add controller –api-version=jakubbujny.com/v1alpha1 –kind=JenkinsPipeline

We need field in our API definition to define microservice name for which pipeline should be created – let’s modify following file: pkg/apis/jakubbujny/v1alpha1/jenkinspipeline_types.go

type JenkinsPipelineSpec struct {
Microservice string `json:”microservice”`
}

now we must regenerate APIs definition so yaml configuration in deploy directory will be the same as our code. To do that we should issue command:

operator-sdk generate openapi

and the final and most important part is to write code in Go which will integrate with Jenkins – to make such integration we need to generate API token in Jenkins and pass it to the operator. I simply made that by environment variables in deploy/operator.yaml

Let’s go to the pkg/controller/jenkinspipeline/jenkinspipeline_controller.go – I will describe only the most important part.

func (r *ReconcileJenkinsPipeline) Reconcile(request reconcile.Request) (reconcile.Result, error) {
	reqLogger := log.WithValues("Request.Namespace", request.Namespace, "Request.Name", request.Name)
	reqLogger.Info("Reconciling JenkinsPipeline")

	// Fetch the JenkinsPipeline instance
	instance := &jakubbujnyv1alpha1.JenkinsPipeline{}
	err := r.client.Get(context.TODO(), request.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}

	resp, err := getSeedJob()

	if err != nil {
		reqLogger.Error(err, "Failed to get seed config to check whether job exists")
		return reconcile.Result{}, err
	}

	if resp.StatusCode == 404 {
		reqLogger.Info("Seed job not found so must be created for microservice "+instance.Spec.Microservice)

		resp, err := createSeedJob()
		err = handleResponse(resp, err, reqLogger, "create seed job")
		if err != nil {
			return reconcile.Result{}, err
		}

		resp, err = updateSeedJob(instance.Spec.Microservice)
		err = handleResponse(resp, err, reqLogger, "update seed job")
		if err != nil {
			return reconcile.Result{}, err
		}
	} else if resp.StatusCode == 200 {
		reqLogger.Info("Seed job found so must be updated for microservice "+instance.Spec.Microservice)
		resp, err = updateSeedJob(instance.Spec.Microservice)
		err = handleResponse(resp, err, reqLogger, "update seed job")
		if err != nil {
			return reconcile.Result{}, err
		}
	} else {
		err = coreErrors.New(fmt.Sprintf("Received invalid response from Jenkins %s",resp.Status))
		reqLogger.Error(err, "Failed to get seed config to check whether job exists")
		return reconcile.Result{}, err
	}

	resp, err = triggerSeedJob()
	err = handleResponse(resp, err, reqLogger, "trigger seed job")
	if err != nil {
		return reconcile.Result{}, err
	}

	return reconcile.Result{}, nil
}
func handleResponse( resp *http.Response, err error, reqLogger logr.Logger, action string) error {
	if err != nil {
		reqLogger.Error(err, "Failed to "+action)
		return err
	}

	if resp == nil {
		return nil
	}

	if resp.StatusCode != 200 {
		err = coreErrors.New(fmt.Sprintf("Received invalid response from Jenkins %s",resp.Status))
		reqLogger.Error(err, "Failed to"+action)
		return err
	}
	return nil
}

func decorateRequestToJenkinsWithAuth(req *http.Request) {
	jenkinsApiToken := os.Getenv("JENKINS_API_TOKEN")
	req.Header.Add("Authorization", "Basic "+ b64.StdEncoding.EncodeToString([]byte("admin:"+jenkinsApiToken)))
}

func getSeedJob() (*http.Response, error) {
	req, err := http.NewRequest("GET", os.Getenv("JENKINS_URL")+"/job/seed/config.xml", nil)
	if err != nil {
		return nil, err
	}
	decorateRequestToJenkinsWithAuth(req)
	return (&http.Client{}).Do(req)
}

func createSeedJob() (*http.Response, error) {
	seedFileData, err := ioutil.ReadFile("/opt/seed.xml")

	req, err := http.NewRequest("POST", os.Getenv("JENKINS_URL")+"/createItem?name=seed", bytes.NewBuffer(seedFileData))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-type", "text/xml")
	decorateRequestToJenkinsWithAuth(req)
	return (&http.Client{}).Do(req)
}

func updateSeedJob(microservice string) (*http.Response, error) {
	resp, err := getSeedJob()
	if err != nil {
		return nil, err
	}
	buf := new(bytes.Buffer)
	_, err = buf.ReadFrom(resp.Body)
	if err != nil {
		return nil, err
	}
	seedXml := buf.String()

	r := regexp.MustCompile(`<defaultValue>(.+)<\/defaultValue>`)
	foundMicroservices := r.FindStringSubmatch(seedXml)

	toReplace := ""
	if strings.Contains(foundMicroservices[1], microservice) {
		return nil,nil
	} else {
		if foundMicroservices[1] == "default" {
			toReplace = microservice
		} else {
			toReplace = foundMicroservices[1] + "," + microservice
		}
	}

	toUpdate := r.ReplaceAllString(seedXml, fmt.Sprintf("<defaultValue>%s</defaultValue>", toReplace))

	req, err := http.NewRequest("POST", os.Getenv("JENKINS_URL")+"/job/seed/config.xml", bytes.NewBuffer([]byte(toUpdate)))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-type", "text/xml")
	decorateRequestToJenkinsWithAuth(req)
	return (&http.Client{}).Do(req)
}

func triggerSeedJob() (*http.Response, error) {
	req, err := http.NewRequest("POST", os.Getenv("JENKINS_URL")+"/job/seed/buildWithParameters", nil)
	if err != nil {
		return nil, err
	}
	decorateRequestToJenkinsWithAuth(req)
	return (&http.Client{}).Do(req)
}

Reconcile function will be triggered when state in Kubernetes must be synced so usually when new object is created. We start with getSeedJob() in line 19 and that function make request to Jenkins to check if seed job already exists – if not (404 code) it’s created with default config located in build/seed.xml and added to Operator’s docker image in build/Dockerfile.

If seed job already exists it must be updated to add microservice name to list of parameters what is done by using regular expressions over seed.xml job config to change default value for parameter.

After all program triggers seed job so it will be executed and new pipeline will be created. Now we should build and push operator docker image and then deploy it

operator-sdk build digitalrasta/jenkins-pipeline-operator

docker push digitalrasta/jenkins-pipeline-operator:latest

kubectl apply -f deploy

And now we can create pipeline for microservice by applying following resource

apiVersion: "jakubbujny.com/v1alpha1"
kind: "JenkinsPipeline"
metadata:
  name: "microservice1"
spec:
  microservice: "microservice1"

The one weak thing is we didn’t create Finalizer for JenkinsPipeline resource – it means after deletion of resource seed’s job parameter won’t be modified so pipeline for that microservice will still exists but Finalizer is topic for another article.

NetworkPolicy on Kubernetes – how to setup properly ingress and egress if I want to limit POD’s network access

WordPress-firewall

Overview

I decided to write this post as I’ve seen on Stackoverflow many posts where people are confused about setting properly NetworkPolicy in Kubernetes – especially how to setup egress to not block traffic which will be sent back to client. I understand that might be confusing especially in TCP protocol where client’s port on which data will be sent back is chosen randomly.

So in that article I will show you how you can setup Minikube to support network policies as out-of-the-box network driver in Minikube doesn’t support that. After some playing with policies we will dig deeper to see how such firewall is implemented in Calico.

As always source code for that article can be found on https://github.com/jakubbujny/article-kubernetes-network-policies

NetworkPolicy in theory

NetworkPolicy allows us to define network rules inside Kubernetes cluster and it’s based on podSelector so it means that we can attach NetworkPolicy to pods matching them using e.g. labels. Those policies can limit outgoing and incoming network access using source/dest ports, CIDRs, namespaces and other pod’s labels.

NetworkPolicy in Kubernetes is only API definition which must be then implemented by network CNI plugin. It means if we define NetworkPolicy and we don’t have in our cluster proper CNI plugin which implements it, that NetworkPolicy won’t have any effect so we will have false-security feeling.

NetworkPolicy in practice – setup Calico on Minikube

Minikube out-of-the-box doesn’t support NetworkPolicies. To use that we must install some external CNI.

One of the most popular and stable plugin is now Calico – it use things which exist in IT for many year like netfilter/iptables and Border Gateway Protocol

To allow Minikube to CNI we must start it with following command:

 minikube start --network-plugin=cni 

And then install Calico by running

 curl https://docs.projectcalico.org/master/manifests/calico.yaml | kubectl apply -f - 

We should see new pods created in kube-system namespace – we should wait until they are in Running state before proceeding:

➜ ~ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7f68846cd-8f97d 0/1 Pending 0 6s
calico-node-mbdzl 0/1 Init:0/3 0 6s

So wait until it will be:

➜ ~ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-7f68846cd-8f97d 1/1 Running 0 74s
calico-node-mbdzl 1/1 Running 0 74s

Prepare pods

As our example application we will use:

  • pod with ubuntu which will have limited ingress to port 80 and egress to Internet access but only on 443 port to restrict that pod to use TLS, named: ubuntu1
  • pod with ubuntu without any limitations to test connectivity, named: ubuntu2

Let’s start with first pod

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu1
  labels:
    app: ubuntu1
spec:
  containers:
    - name: ubuntu
      image: ubuntu:18.04
      command: ["bash", "-c"]
      args: ["apt-get update && apt-get install -y curl python3 && while true; do sleep 1; done"]
      ports:
        - containerPort: 80

That’s simple pod which in start command install curl for testing and python3 for starting simple web server in next steps. After all it goes into infinite loop so we can exec into that container and write some commands.

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu2
  labels:
    app: ubuntu2
spec:
  containers:
    - name: ubuntu
      image: ubuntu:18.04
      command: ["bash", "-c"]
      args: ["apt-get update && apt-get install -y curl && while true; do sleep 1; done"]

Second pod looks the same as first but we don’t install python and don’t expose port.

Network communication – sysdig

In current communication network is not limited – let’s verify it and see how looks network traffic.

To achieve that we will use sysdig – tool which is preinstalled on Minikube machine and it’s like strace combined with tcpdump but much more powerful. Minikube underhood spawn virtual machine where Kubernetes is installed. We can SSH into that machine and make ourselves root

minikube ssh

sudo su –

after that we want to observe traffic on port 80. We can do it using following command:

sysdig fd.port=80

Now terminal should freeze, waiting for traffic to dump.

Let’s open another terminal and find ubuntu1 IP address

kubectl describe pod ubuntu1

….

Status: Running
IP: 192.168.120.77
Containers:

exec into ubuntu1 pod and start web server

kubectl exec -it ubuntu1 bash

python3 -m http.server 80

Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) …

open another terminal and exec into ubuntu2 pod, send HTTP request to ubuntu1 webserver

kubectl exec -it ubuntu2 bash
root@ubuntu2:/# curl 192.168.120.77

Now in sysdig terminal we should see many new logs lines, look on the first

80565 10:16:48.740561399 0 curl (11292) 192.168.120.77:80

as we can see ubuntu2 opened connection to ubuntu1:80 – randomly chosen port to send back data is 37454

Network policies – ingress and egress

Now it’s time to limit the network for ubuntu1 pod. There will be 3 policies:

  • UDP egress on port 53 – that allow DNS traffic so we can translate e.g. google.com to ip address inside pod
  • TCP ingress on port 80 – allow clients to connect to our webserver
  • TCP egress on port 443 – allow pod to connect to Internet services on TLS port

Configuration:

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dns
spec:
  podSelector:
    matchLabels:
      app: ubuntu1
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: UDP
          port: 53

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-webserver
spec:
  podSelector:
    matchLabels:
      app: ubuntu1
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - port: 80
          protocol: TCP

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-tls
spec:
  podSelector:
    matchLabels:
      app: ubuntu1
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - port: 443
          protocol: TCP

Now we can go back to ubuntu1’s console and try to open connection to google.com on port 80 – we should see timeout and on HTTPS connection we should see response.

root@ubuntu1:/# curl -m 5 -vv google.com
* Rebuilt URL to: google.com/
* Trying 216.58.215.78…
* TCP_NODELAY set
* Connection timed out after 5004 milliseconds
* stopped the pause stream!
* Closing connection 0
curl: (28) Connection timed out after 5004 milliseconds

root@ubuntu1:/# curl https://google.com

Now let’s start again webserver and go back to ubuntu2 to test connection again. We still see answer from webserver.

So how is that possible? We defined limiting rule for egress traffic so we see that traffic is only possible on port 443 but data is somehow still sent back to client on some other port.

Deep dive – Calico iptables

Firewall rules which we defined must be then implemented by CNI. Calico use netfilter/iptables for that – every time we define NetworkPolicy, Calico automatically setup proper netfilter rules on every node in cluster.

Let’s go back onto minikube VM and by making some combination of iptables -S and grep try to find related rules.

-A cali-pi-_Pyx9r8CS7bPqC0nMrCi -p tcp -m comment –comment “cali:x8PiQWJp-yhKM8vP” -m multiport –dports 80 -j MARK –set-xmark 0x10000/0x10000

-A cali-tw-cali22716d29a85 -m comment –comment “cali:MfYo–qV2fDDHXqO” -m mark –mark 0x0/0x20000 -j cali-pi-_Pyx9r8CS7bPqC0nMrCi

-A cali-from-wl-dispatch -i cali22716d29a85 -m comment –comment “cali:8gmcTnib5j5lzG4A” -g cali-fw-cali22716d29a85

-A cali-fw-cali22716d29a85 -m comment –comment “cali:CcU6YKJiUYOoRnia” -m conntrack –ctstate RELATED,ESTABLISHED -j ACCEPT

As we can see Calico defines custom tables and targets – we don’t want to do whole reverse engineering of that as it’s really complex.

In simple words, in first line, we see our main iptables rule on port 80 which make MARK on the packets – MARK means advanced routing rules which are defined by Calico.

Second line show us how Calico define relations inside those iptables rules so id Pyx9r8CS7bPqC0nMrCi is related to cali22716d29a85.

In third line we see that cali22716d29a85 is actually network interface defined on node and packets processing chain should go to cali-fw-cali22716d29a85.

Finally the most important fourth line has –ctstate RELATED,ESTABLISHED -j ACCEPT parameters. Netfliter is stateful firewall – it means it understands how TCP connection works and can track in memory which connections are new and which are related or  established. So if netfilter see that some client on port 37454 already established connection to port 80 it will track that connection and won’t drop packets by egress rule limiting network traffic to only 443 port. The same rule applies when pod open connection to some Internet services so ingress rule won’t drop packets.

 

Immutable Single Page Application – wrap frontend with docker and reuse it on many environments

Problem overview

Docker conquer the world – the basic questions it how to place frontend application inside container? In docker world we want to build artifact (image) once and reuse it on many environments. Such immutability of images gives us some guarantees regarding consistency of environments so our staging can be as much as possible similar to production – that’s good for stability of our system because we can be sure that code deployed on staging will behave the same on production. So to be able to run one artifact in different environments we usually extract whole env-specific configuration to environment variables or some ConfigMaps which are mounted in runtime as config files and contains information required to run on particular environment e.g. db connection, secrets, public domain name etc.

In case of backend services it’s really simple as they can read environment variables in runtime which are created by DevOps team but what about frontend? Here situation is not so trivial as usually we can pass environment variables to frontend only in build time and those variables are just placed as a strings in static js files (e.g in React). It means that we need separate build pipeline of frontend for different environments like frontend-development, frontend-staging, frontend-production, frontend-client1, etc. So in such situation we must have many different artifacts of one codebase what is not really good situation – please visit site https://immutablewebapps.org/ if you want to know more.

Implementation of solution

We need possibility to configure frontend in runtime – it means before starting static files hosting we must configure frontend files so they can work well with environment on which they are hosted at the moment. When using docker that’s really easy – full source code explained in this article can be found there: https://github.com/jakubbujny/article-immutable-single-page-application

Let’s say that we have simple SPA application which is divided into index.html file as entrypoint and script.js where whole site is located. Common case is to inject URL to backend API which can be located in different places in different situations, e.g.:

  • locally I want to use http://localhost:port as API URL
  • maybe I want to test frontend on mobile which is in the same LAN as my PC so I want to inject http://my_pc_ip:port as API URL
  • maybe I want to create Review Apps so my API URL will look like api-git_commit_hash.review.my-project.com
  • on normal envs I just want to use api.env.myproject.com

As example we can use such simple html file which just make place to show API URL and include our dummy script:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<div >I'm gonna make shots to <span id="foo" style="color: red"> </span> </div>
<script src="script.js"> </script>
</body>
</html>

Source code of script.js

window.onload = function (ev) {
    var element = document.getElementById("foo")
    element.innerHTML = ? //of course that won't work - we must think what should we place here
}

Dockerfile would look like:

FROM nginx:1.15.7

ADD index.html /usr/share/nginx/html/index.html

ADD script.js /usr/share/nginx/html/script.js

CMD nginx -g "daemon off;"

So we must consider how to pass environment variable to static files when container is starting. The simplest solution is to use such trick in CMD section of Dockerfile:

CMD sed -i "0,/#API_URL_TOKEN#/{s@#API_URL_TOKEN#@${API_URL}@}" /usr/share/nginx/html/index.html && nginx -g "daemon off;"

This sed command look for the first (0,) occurrence of #API_URL_TOKEN# in index.html file and then replace it with API_URL environment variable. Usage of “@” as delimiter in sed command is very important as when we will use standard “/” we have conflict with protocol part of url (https://). After such configuration nginx is starting.

Then in section of index.html we should add following script:

<script>
        apiUrl = "#API_URL_TOKEN#"
        if (apiUrl === "#API_URL_TOKEN#") {
            apiUrl = "localhost"
        }
</script>

So just place global variable with apiUrl and token as value. As we replace only first occurrence we can write condition: if replace not happened just set defaults for local development.

Now we can just place

element.innerHTML = apiUrl

and use following commands to build and test our POC

docker build -t jakubbujny/immutable-spa .
docker run -p 80:80 -e API_URL=https://injected-in-env.jakubbujny.com -it jakubbujny/immutable-spa

Those commands just build docker image and run it with API_URL env variable which will be injected into index.html file. Effect:
Screenshot from 2018-12-26 12-47-21

Dedicated to O.S. 😉

GitOps – declarative Continuous Deployment on Kubernetes in simple steps

Intro

In this article I’m going to show you very basic and simple example which describes GitOps concept – new pattern in DevOps world which helps to keep infrastructure stable and more recoverable. By using git as operations tool we can see the whole history of changes and we are able to say who and when changed something. As example I want to show you really simple Continuous Deployment pipeline of docker images on Kubernetes cluster.

Main assumptions are:

  • Git repository is single source of truth about state of the system
  • If state change in repository it will be automatically synchronized

Proposed architecture:

Article - gitops cd

Deployment design

Let’s say that we have small company with 3-4 projects managed by single DevOps team on the same infrastructure and Kubernetes cluster. Every project is based on microservices architecture with 4-5 services. Every project should have at least 2-3 environments:

  • Development – with CD pipeline from development branch of every microservice
  • Staging/Production – stable environment where microservices are deployed only on-demand when management says that version is stable, well tested and ready to handle clients. In our example from implementation point of view staging and production are the same.

Docker images

To keep simple mapping between the source code and artifact which will be deployed on some environment, docker images should be tagged using git commit hash – that’s ID which allows us to simply identify which code is deployed so in case of debugging some complex environments that’s really helpful. We can achieve that easily in CI pipeline which produce docker images by using script like:

HEAD=$(git rev-parse HEAD)

docker build -t registry.gitlab.com/myproject/microservice:${HEAD} .

docker push registry.gitlab.com/myproject/microservice:${HEAD}

Environments state repository

Heart of GitOps concept is state repository which should be source of truth for whole system. It means that in case of failure we should be able to recover the system using configuration stored in git repository. In our example we will use simpler concept – store only git hash of particular microservice what will represent version of application deployed on particular environment. It should allow us to make our infrastructure really reproducible but not on 100% as deployment configuration like environment variables can change behavior of the system. It means that in real GitOps pattern we should also store in such repository configuration of deployment like kubernetes yaml files, infrastructure descriptors like terraform and ansible files etc. It’s better but much more complicated as changes pushed to repository should be automatically synchronized by applying the diff.

Repository structure

Screenshot from 2018-12-23 12-26-08

So every deployment has own directory where name of directory is also name of the namespace in Kubernetes cluster. Every microservice in particular deployment has one file where git commit hash is stored. There is also one special file images_path which contains docker image path. Docker image name with tag will be constructed using following pattern:

content of images_path file (e.g. registry.gitlab.com/mycompany/project/development) + deployment file name (e.g. authentication) + “:” + content of deployment file (e.g. 36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2)

Example: registry.gitlab.com/mycompany/project/development/authentication:36a03b3f4c8ba4fc0bdcc529450e557ae08c12f2

There is also one special directory sync-agent which will be described in next paragraph.

Sync agent

To avoid making manual updates on environment it’s required to have some synchronization agent which will read state of repository and apply changes in deployment. To achieve that we will use simple CronJob on Kubernetes which will run periodically and use kubectl to update image of particular deployment. Sync agent will be created per Kubernetes namespace to improve isolation between environments. So at first we must create proper RBAC permissions to allow our job to update deployments.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sync-agent
  namespace: project1-development

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sync-agent
  namespace: project1-development
rules:
- apiGroups: ["extensions"]
  resources: ["deployments"]
  verbs: ["get","list","patch","update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sync-agent
  namespace: project1-development
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sync-agent
subjects:
- kind: ServiceAccount
  name: sync-agent
  namespace: project1-development

In above configuration we create service account for sync-agent and give to it permissions to manipulate deployments within his namespace.

Next important thing is to create secret with deploy key – such deploy key should be configured to have read-only access to state repository so sync agent can clone the repo and read deployment files.

apiVersion: v1
kind: Secret
metadata:
  name: sync-agent-deploy-key
  namespace: project1-development
type: Opaque
data:
  id_rsa: 

Final part is to create sync-agent CronJob which will run every minute and make synchronization:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: sync-agent
  namespace: project1-development
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 30
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sync-agent
          containers:
          - name: sync-agent
            image: lachlanevenson/k8s-kubectl:v1.10.3
            command:
            - /bin/sh
            - -c
            args:
            - apk update &&
              apk add git &&
              apk add openssh &&
              mkdir -p /root/.ssh && cp /root/key/id_rsa /root/.ssh/id_rsa &&
              chmod 0600 /root/.ssh/id_rsa &&
              ssh-keyscan -t rsa gitlab.com >> /root/.ssh/known_hosts &&
              mkdir -p /deployment && cd /deployment &&
              git clone git@gitlab.com:my-project/environments-state.git &&
              cd environments-state/sync-agent &&
              NAMESPACE=project1-development ./sync-repository-and-kubernetes-state.sh
            volumeMounts:
            - name: sync-agent-deploy-key
              mountPath: /root/key/
          volumes:
          - name: sync-aginfrastructure ent-deploy-key
            secret:
              secretName: sync-agent-deploy-key
          restartPolicy: Never

Args and command section are a little hacked as we just use public image with kubectl we must install additional things at beginning of the container to keep example simple – probably in real usage we should create docker image with pre-installed tools and push it to some private registry.

Comments to lines:
7 – run job every minute
22-24 – install required tools
25-26 – ssh private key is mounted as file but we must copy it to proper place and set proper chmod to be able to use it
27 – add gitlab.com to known hosts file so git will be able to clone repo
29-31 – clone repository and cd to sync-agent dir where script sync-repository-and-kubernetes-state.sh is located

Code of the sync script:

#!/usr/bin/env sh
set -e
set -x

if [ -z ${NAMESPACE+x} ]; then echo "NAMESPACE env var is empty! Cannot proceed"; exit 1; fi

cd ../$NAMESPACE

IMAGES_PATH=$(cat images_path)
for deployment_file in *.deployment; do
    DEPLOYMENT_HASH=$(cat $deployment_file)
    DEPLOYMENT_NAME=$(echo $deployment_file | sed 's/\.deployment//')
    kubectl set image --namespace $NAMESPACE deployment/$DEPLOYMENT_NAME $DEPLOYMENT_NAME=${IMAGES_PATH}/${DEPLOYMENT_NAME}:${DEPLOYMENT_HASH}
done

Code is really simple – script takes NAMESPACE as parameter and then iterate over all deployment files making kubectl set image on every deployment. Kubernetes will do nothing when you try to set image with the same value which is already in deployment so making such set many times is idempotent. When some new image hash appears kubectl set image cause rolling upgrade on deployment.

Deployment procedure

Whole infrastructure is in place so now to create CD pipeline from development branch we must just place code which will:

  • read current git HEAD hash
  • tag docker with git HEAD hash and push it
  • commit to state repository new hash to file and push it

After such actions we can expect in around 1 minute that sync-agent will clone new state and make synchronization on Kubernetes cluster.

To make on-demand deployment on some stable environment we can just clone repository on our PC and place proper hashes in files manually and again sync-agent will do the work for us.

 

Container and cluster scaling on Kubernetes using Horizontal Pod Autoscaler and Cluster Autoscaler on AWS EKS

Introduction

Kubernetes is about scaling but it doesn’t mean that we have auto scaling out-of-the-box – we must activate some additional components and configure them. In this article I want to show you working example of simple service scaling on Kubernetes using Horizontal Pod Autoscaler. After reaching maximum cluster capacity we will automatically add more workers to our cluster using Cluster Autoscaler. Everything will run on Amazon EKS – managed Kubernetes on AWS.

article-kubernetes-scaling

As always full source code can be found on my github: https://github.com/jakubbujny/article-scale-containers-and-eks

Simple deployment creating load

To test auto scaling we need some simple service which will create big load for us so scaling can be triggered. For that task we are going to use following Python code:

import flask
import uuid
import hashlib
app = flask.Flask(__name__)
@app.route("/")
def hello():
    for i in range(0,800000):
      hashlib.sha224(uuid.uuid4().hex.upper()[0:6].encode()).hexdigest()
    return "Done"

app.run(host="0.0.0.0", threaded=True)

That code is really simple – just create web endpoint using Flask framework – GET request on “/” will cause long loop which calculate a lot of SHA hashes from random UUID what should take about 5–10 seconds and consume a lot of CPU during that time.

To avoid building own Docker image as we want to avoid creating docker registry (to simplify example) we can use simple trick by taking docker image jazzdd/alpine-flask:python3 which is available on dockerhub and contains Python/Flask installed. So we can create our python file in “command” section and run it, see full yaml below:

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: default
  name: microservice
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: microservice
    spec:
      containers:
      - name: microservice
        image: jazzdd/alpine-flask:python3
        command: ["sh"]
        args: ["-c", "printf \"import flask\\nimport uuid\\nimport hashlib\\napp = flask.Flask(__name__)\\n@app.route(\\\"/\\\")\\ndef hello():\\n    for i in range(0,800000):\\n     hashlib.sha224(uuid.uuid4().hex.upper()[0:6].encode()).hexdigest()\\n    return \\\"Done\\\"\\napp.run(host=\\\"0.0.0.0\\\", threaded=True)\" > script.py && python3 script.py"]
        ports:
        - name: http-port
          containerPort: 5000
        resources:
          requests:
            cpu: 200m

Important thing there is resources request block which says that on 1 CPU core machine (which we are going to use in this article) we can create 5 microservice PODs (200m x 5 = 1000m = 1CPU) and reaching that number means end of capacity of particular node. Reaching cluster capacity will be trigger for Cluster Autoscaler.

Horizontal Pod Autoscaler

Horizontal scaling in Kubernetes world means adding more pods in particular deployment. To achieve that Horizontal Pod Autoscaler can be used but we need to note one important thing: In the newest Kubernetes version metrics-server need to be installed to use HPA – Heapster is deprecated and shouldn’t be used anymore. 

To test that on minikube you need just to type:

minikube addons enable metrics-server

To deploy metrics-server on EKS you need to clone following repository: https://github.com/kubernetes-incubator/metrics-server and then issue command:

kubectl apply -f metrics-server/deploy/1.8+/

To activate HPA for our microservice we need to apply following yaml file:

---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  namespace: default
  name: microservice
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: microservice
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 50

targetAverageUtilization: 50 means that Kubernetes will try to maintain half of usage of CPU requested by our microservice (50% * 200m = 100m) on particular POD. E.g. when we have single POD which is having 200m of CPU, Kubernetes will create new POD so 200m can be divided on 2 PODs (100m and 100m).

AWS EKS and Cluster Autoscaler

Disclaimer – why use Cluster Autoscaler instead of ASG scaling trigger based on CPU?

From Cluster Autoscaler FAQ:

“Cluster Autoscaler makes sure that all pods in the cluster have a place to run, no matter if there is any CPU load or not. Moreover, it tries to ensure that there are no unneeded nodes in the cluster.

CPU-usage-based (or any metric-based) cluster/node group autoscalers don’t care about pods when scaling up and down. As a result, they may add a node that will not have any pods, or remove a node that has some system-critical pods on it, like kube-dns. Usage of these autoscalers with Kubernetes is discouraged.”

For EKS deployment we are going to use modified EKS version from my previous article.

Cluster Autoscaler is component which will be installed on EKS cluster. It will look in Kubernetes API and make request to AWS API to scale worker nodes’s ASG. It means that node on which Cluster Autoscaler will reside need proper IAM policy which will allow container from that node to make operations on ASG.

 resource "aws_iam_role_policy" "for-autoscaler" {
  name = "for-autoscaler"
  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:DescribeTags",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}
POLICY
  role = "${aws_iam_role.eks-node.name}"
}

That policy should be probably limited in Resource section but we will leave * to simplify example.

We put some additional tags to ASG to use them in Cluster Autoscaler

  tag {
    key = "k8s.io/cluster-autoscaler/enabled"
    value = "whatever"
    propagate_at_launch = false
  }

  tag {
    key                 = "kubernetes.io/cluster/eks"
    value               = "owned"
    propagate_at_launch = true
  }

We must also setup security groups to allow 443 port communication from cluster control plane to nodes as mentioned in this issue: https://github.com/kubernetes-incubator/metrics-server/issues/45

For Cluster Autoscaler we will modify a little example deployment from here: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

We need to modify tags which Cluster Autoscaler will use to discover ASG which will be scaled:

- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,kubernetes.io/cluster/eks

Add env with region in which we are operating:

 env:
   - name: "AWS_REGION"
     value: eu-west-1

Change certificate to use what is required for EKS:

 volumeMounts:
   - name: ssl-certs
     mountPath: /etc/ssl/certs/ca-bundle.crt
     readOnly: true

Cluster Autoscaler is ready to use and will scale up or down worker nodes using ASG scaling between 1 and 10 instances.

Testing

Last step is to create load balancer attached to microservice and test auto scaling by making some requests to create load.

apiVersion: v1
kind: Service
metadata:
  name: microservice
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  labels:
    app: microservice
spec:
  ports:
  - port: 80
    targetPort: http-port
  selector:
    app: microservice
  type: LoadBalancer

You can just simply try to open load balancer endpoint on root in web browser and hit f5 a few times to generate load or use script like:

while true; do sleep 2; timeout 1 curl http://<elb_id>.elb.eu-west-1.amazonaws.com/; done

 

After that you should see in Kubernetes that HPA scaled your containers up and reached maximum node capacity. After while Cluster Autoscaler should scale AWS ASG and add new worker node so HPA can complete PODs scaling.

Replicate cloud AWS RDS MySQL to on-premise PostgreSQL in Docker – future is today! Debezium and Kafka on AWS EKS

Introduction

Nowadays applications have bigger requirements then older ones – many SaaS systems need to operate globally on all continents or in hybrid solutions, sharing some data between private and public clouds. That requirement is really complicated as many systems and users require real-time responses – there are expectations that modern IT systems are fast, efficient and provide zero waiting time. Looking on pure speed of light (as in ideal environment that’s speed of electrons running through cables) when some information need to pass 10 000 km it will take around 30ms and that’s limit coming from physics (best regards Einstein!). We all know that it will take much longer because of electronic parts delays, routers, protocols overhead, traffic congestion, etc. How to deal with that?

As currently IT systems are more about reads than writes (probably not in IoT but that’s different story) there is concept to replicate data from “source of truth” and store it near to client to provide following advantages:

  • data is near client so he can make fast reads
  • when “source of truth” database is down, reads from replica are still working
  • when connection with “source of truth” is broken, reads from replica are still working
  • load on “source of truth” can be reduced by passing all reads to replica

In this article I show you how to replicate changes happening in database located in AWS cloud to database located on-prem. We can achieve that using Debezium tool which use CDC concept (change data capture) to stream changes onto Kafka, acting as data producer. On the other side we can attach Kafka Connect tool which can consume that stream and write that to target database using JDBC. To make it more interesting and easier I deployed Kafka and Debezium on Amazon EKS – new Kubernetes service in AWS. See architecture diagram:

article-rds-debezium-postgres-kafka-kubernetes (2)

As always full source code can be found on my Github:

https://github.com/jakubbujny/article-replicate-cloud-AWS-RDS-MySQL-to-on-premise-PostgreSQL-in-Docker-future-is-today

Disclaimer – security

Configuration showed in that article is definitely far away from proper security standards required by production systems looking on e.g. lack of SSL/encryption, free public access to services, not limited network or weak passwords so please keep in mind that it’s more about POC less about implementation which you can copy/paste to your real/production system.

Configure AWS EKS using Terraform

There is perfect tutorial describing EKS deployment using Terraform: https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html As EKS availability is limited for now I chose Ireland region and modified a little configuration from tutorial.

EKS master configuration is very easy and require a few blocks:


resource "aws_iam_role" "eks-cluster" {
  name = "eks-cluster"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = "${aws_iam_role.eks-cluster.name}"
}

resource "aws_iam_role_policy_attachment" "AmazonEKSServicePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = "${aws_iam_role.eks-cluster.name}"
}

resource "aws_eks_cluster" "eks" {
  name            = "eks"
  role_arn        = "${aws_iam_role.eks-cluster.arn}"

  vpc_config {
    security_group_ids = ["${aws_security_group.eks-cluster.id}"]
    subnet_ids         = ["${aws_subnet.eks_a.id}","${aws_subnet.eks_b.id}"]
  }

  depends_on = [
    "aws_iam_role_policy_attachment.AmazonEKSClusterPolicy",
    "aws_iam_role_policy_attachment.AmazonEKSServicePolicy",
  ]
}

Important notes to lines:

21,26 – AWS delivered proper policies for us, we need only to attach them

30 – Create EKS cluster (masters) and place in defined network. That operation may take 10-20 minutes so be calm.

And that's almost all what is required to create EKS masters – the last thing which we need is kubectl config which is required to administrate Kubernetes. To obtain config we will use following Terraform output:

locals {
  kubeconfig = <<KUBECONFIG
apiVersion: v1
clusters:
- cluster:
    server: ${aws_eks_cluster.eks.endpoint}
    certificate-authority-data: ${aws_eks_cluster.eks.certificate_authority.0.data}
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "eks"
KUBECONFIG
}

output "kubeconfig" {
  value = "${local.kubeconfig}"
}

There is one important thing there – aws-iam-authenticator command will be executed on your PC. It means that you need to go on https://github.com/kubernetes-sigs/aws-iam-authenticator/releases download cli and place it somewhere in your PATH e.g. in /usr/bin with name aws-iam-authenticator. After that execute following command to get your kubectl config:

 terraform output kubeconfig > kubeconfig.yaml

Adding nodes to EKS cluster is a little more complex.

Again attach policies delivered by AWS to role.

resource "aws_iam_role" "eks-node" {
  name = "eks-node"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "eks-node-AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = "${aws_iam_role.eks-node.name}"
}

resource "aws_iam_role_policy_attachment" "eks-node-AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = "${aws_iam_role.eks-node.name}"
}

resource "aws_iam_role_policy_attachment" "eks-node-AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = "${aws_iam_role.eks-node.name}"
}
resource "aws_iam_instance_profile" "eks-node" {
name = "eks-node"
role = "${aws_iam_role.eks-node.name}"
}

Next create Auto Scaling Group to get up Kubernetes node:

data "aws_ami" "eks-worker" {
  filter {
    name   = "name"
    values = ["amazon-eks-node-v*"]
  }

  most_recent = true
  owners      = ["602401143452"] # Amazon Account ID
}

locals {
  eks-node-userdata = <<USERDATA
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh --apiserver-endpoint '${aws_eks_cluster.eks.endpoint}' --b64-cluster-ca '${aws_eks_cluster.eks.certificate_authority.0.data}' 'eks'
USERDATA
}

resource "aws_launch_configuration" "node" {
  associate_public_ip_address = true
  iam_instance_profile        = "${aws_iam_instance_profile.eks-node.name}"
  image_id                    = "${data.aws_ami.eks-worker.id}"
  instance_type               = "t2.medium"
  name_prefix                 = "terraform-eks"
  security_groups             = ["${aws_security_group.eks-node.id}"]
  user_data_base64            = "${base64encode(local.eks-node-userdata)}"

  root_block_device {
    delete_on_termination = true
    volume_size = 20
    volume_type = "gp2"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "node" {
  desired_capacity     = 1
  launch_configuration = "${aws_launch_configuration.node.id}"
  max_size             = 1
  min_size             = 1
  name                 = "eks"
  vpc_zone_identifier  = ["${aws_subnet.eks_a.id}"]

  tag {
    key                 = "Name"
    value               = "eks"
    propagate_at_launch = true
  }

  tag {
    key                 = "kubernetes.io/cluster/eks"
    value               = "owned"
    propagate_at_launch = true
  }
}

Important notes to lines:

1 – fetch AMI created by AWS which can be used as EKS node. AWS installed there proper scripts/configuration.

11 – command in user data for instance required by EKS to configure node and things like kubelet, systemd service etc.

23 – t2.medium should be enough to run Kafka (Zookeeper + Broker) and Debezium

30 – we need at least 20G disk as AMI has 20G size

39 – create ASG which will spawn one node for us

So as last step we need to apply following ConfigMap to allow nodes to join cluster, generated by Terraform:

locals {
  config-map-aws-auth = &lt;<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span><span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>&lt;CONFIGMAPAWSAUTH

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: ${aws_iam_role.eks-node.arn}
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
CONFIGMAPAWSAUTH
}

output &quot;config-map-aws-auth&quot; {
  value = &quot;${local.config-map-aws-auth}&quot;
}

And create service account with admin rights to use in Kubernetes dashboard:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: eks-admin
  namespace: kube-system

To get admin token which can be used in Kubernetes Dashboard login we can use following command:

kubectl -n kube-system describe secret $(./kubectl.sh -n kube-system get secret | grep eks-admin | awk '{print $1}') | grep token

To get full source code of those configs please take a look on: https://github.com/jakubbujny/article-replicate-cloud-AWS-RDS-MySQL-to-on-premise-PostgreSQL-in-Docker-future-is-today/tree/master/aws

Configure AWS RDS using Terraform

We need to configure Amazon RDS as our “source of truth” – database which we will replicate to on-prem deployment. Debezium will read binlog from MySQL to stream changes on Kafka.

resource "aws_db_instance" "rds" {
  apply_immediately = true
  publicly_accessible = true
  skip_final_snapshot = true
  identifier = "mymysql"
  availability_zone = "${local.region}a"
  allocated_storage    = 5
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "5.7"
  instance_class       = "db.t2.micro"
  name                 = "mydb"
  username             = "mymysql"
  password             = "mysqlmysql"
  db_subnet_group_name = "${aws_db_subnet_group.rds.name}"
  vpc_security_group_ids = ["${aws_security_group.rds.id}"]
  backup_retention_period = 1
  parameter_group_name = "${aws_db_parameter_group.rds.name}"
}

resource "aws_db_parameter_group" "rds" {
  family = "mysql5.7"
  name = "rds"

  parameter {
    name = "binlog_format"
    value = "ROW"
  }
}

Important notes to lines:

3 – make RDS public to easily access DB from our PC – we need to insert some data

15 – we need to create DB subnet group which contains at least 2 subnets even when we create single-AZ database – that’s requirement from AWS coming from possibility of promoting single-AZ deployment to multi-AZ deployment

17 – that’s important – automatic backups enable possibility of streaming binlog which is required by Debezium

21 – we need to attach custom parameter group to stream binlog in ROW format required by Debezium

Deploy Kafka and Zookeeper as StatefulSet on Kubernetes

Kafka requires Zookeeper to work what is well known fact. We can deploy both things as 2 different StatefulSets with AWS EBS attached for data. As we want to keep that POC simple we will just use one Kubernetes node and expose Kafka on node-port to allow on-prem client to connect. For Kafka deployment we will use popular docker images from Wurstmeister.

To allow dynamic provisioning of EBS volumes by Kubernetes we need to define following StorageClass

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zones: eu-west-1a
  fsType: ext4

In Zookeeper Dockerfile we can find following 2 volumes:

VOLUME ["/opt/zookeeper-${ZOOKEEPER_VERSION}/conf", "/opt/zookeeper-${ZOOKEEPER_VERSION}/data"]

So we need to attach EBS volumes to those 2 paths to keep our state safe.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zookeeper
  labels:
    app: zookeeper
spec:
  selector:
    matchLabels:
      app: zookeeper
  serviceName: "zookeeper"
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: zookeeper
        image: wurstmeister/zookeeper:3.4.6
        ports:
        - containerPort: 2181
          name: zookeeper
        volumeMounts:
        - name: zookeeper-data-pv-claim
          mountPath: /opt/zookeeper-3.4.6/data
        - name: zookeeper-conf-pv-claim
          mountPath: /opt/zookeeper-3.4.6/conf
  volumeClaimTemplates:
    - metadata:
        name: zookeeper-data-pv-claim
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: ebs
        resources:
          requests:
            storage: 1Gi
    - metadata:
        name: zookeeper-conf-pv-claim
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: ebs
        resources:
          requests:
            storage: 1Gi

So by defining volumeClaims we say to Kubernetes that he should create dynamically  EBS volumes using AWS API, make ext4 file system and mount them in proper paths. All more what we need is service which will allow to access Zookeeper inside cluster.

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  labels:
    app: zookeeper
spec:
  ports:
  - port: 2181
    name: zookeeper
  clusterIP: None
  selector:
    app: zookeeper

clusterIP: None means that service is Headless service so Kubernetes will expose POD’s IP directly under service’s DNS record.

Kafka deployment is a little tricky because of “advertised listener” – it means that we need to put node’s public IP which will be presented to clients. We cannot easily use e.g. Kubernetes Downward API as status.hostIP will be private node’s IP, not accessible from on-prem infrastructure. So for keeping POC simple we will just fetch nodes’s public IP using query by tag name:eks via AWS CLI.

aws --region eu-west-1 ec2 describe-instances --filters "Name=tag:Name,Values=eks" --filters "Name=instance-state-name,Values=running" --query "Reservations[0].Instances[0].PublicIpAddress"

then inject it to proper environment variable during deployment by sed NODE_PUBLIC_IP in Kafka’s yaml file using following script:

kafka_file=$(cat kafka.yaml | sed -- expression="s/NODE_PUBLIC_IP/$public_node_ip/g")
echo "$kafka_file" | kubectl apply -f -

That simple trick read Kafka deployment file from disk, make sed in the fly and then apply it to Kubernetes cluster by sending changed file on kubectl STDIN.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  selector:
    matchLabels:
      app: kafka
  serviceName: "kafka"
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: kafka
        image: wurstmeister/kafka:2.11-2.0.0
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zookeeper:2181
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://NODE_PUBLIC_IP:30002
        - name: KAFKA_LISTENERS
          value: PLAINTEXT://:9092
        ports:
        - containerPort: 9092
          name: kafka
        volumeMounts:
        - name: kafka-data-pv-claim
          mountPath: /kafka
  volumeClaimTemplates:
    - metadata:
        name: kafka-data-pv-claim
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: ebs
        resources:
          requests:
            storage: 1Gi

So again we attach EBS disk for data on Kafka and we connect Kafka to Zookeeper using service (zookeeper:2181). What we need next is Kafka service so Debezium can access it locally for bootstrap and we need to expose Kafka on NodePort.

apiVersion: v1
kind: Service
metadata:
  name: kafka
  labels:
    app: kafka
spec:
  ports:
  - port: 9092
    name: kafka
  clusterIP: None
  selector:
    app: kafka
---

apiVersion: v1
kind: Service
metadata:
  name: kafka-node
  labels:
    app: kafka-node
spec:
  ports:
  - port: 9092
    targetPort: 9092
    name: kafka
    nodePort: 30002
  type: NodePort
  selector:
    app: kafka

Please note that we expose Kafka on non-standard port 30002 due to default NodePort range 30000-32767

After those operations there should be working deployment of Kafka on Kubernetes wich can be tested by connecting from local PC on node_public_ip:30002

Deploy and configure Debezium on Kubernetes

Debezium is type of Kafka Connector what means that it stores whole state in Kafka so we can deploy it as Kubernetes Deployment because from Kubernetes point of view it’s stateless application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: debezium-connect-source
spec:
  selector:
    matchLabels:
      app: debezium-connect-source
  replicas: 1
  template:
    metadata:
      labels:
        app: debezium-connect-source
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: debezium-connect-source
        image: debezium/connect:0.8
        env:
        - name: BOOTSTRAP_SERVERS
          value: kafka:9092
        - name: GROUP_ID
          value: debezium-connect-source
        - name: CONFIG_STORAGE_TOPIC
          value: debezium-connect-source_config
        - name: OFFSET_STORAGE_TOPIC
          value: debezium-connect-source_offset
        ports:
        - containerPort: 8083
          name: dm-c-source

There are 3 important env variables which are perfectly described on Debezium’s dockerhub site:

GROUP_ID
This environment variable is required when running the Kafka Connect service. Set this to an ID that uniquely identifies the Kafka Connect cluster the service and its workers belong to.

CONFIG_STORAGE_TOPIC
This environment variable is required when running the Kafka Connect service. Set this to the name of the Kafka topic where the Kafka Connect services in the group store connector configurations. The topic must have a single partition and be highly replicated (e.g., 3x or more).

OFFSET_STORAGE_TOPIC
This environment variable is required when running the Kafka Connect service. Set this to the name of the Kafka topic where the Kafka Connect services in the group store connector offsets. The topic must have a large number of partitions (e.g., 25 or 50), be highly replicated (e.g., 3x or more) and should be configured for compaction.

Last step on Kubernetes is to expose Debezium on NodePort so we can easily configure it from PC.

apiVersion: v1
kind: Service
metadata:
  name: debezium-connect-source
  labels:
    app: debezium-connect-source
spec:
  ports:
  - port: 8083
    targetPort: 8083
    name: debezium-connect-source
    nodePort: 30001
  type: NodePort
  selector:
    app: debezium-connect-source

Now is time to configure Debezium – it can be done using web API exposed by Debezium where we need to define connector with proper options. So first configuration json:

{
  "name": "my-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "mymysql.clmru7r0arad.eu-west-1.rds.amazonaws.com",
    "database.port": "3306",
    "database.user": "mymysql",
    "database.password": "mysqlmysql",
    "database.server.id": "184054",
    "database.server.name": "mydebezium",
    "database.whitelist": "mydb",
    "database.history.kafka.bootstrap.servers": "kafka:9092",
    "database.history.kafka.topic": "schema-changes.mydb",
    "transforms": "route",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
    "transforms.route.replacement": "$3"
  }
}

Important notes to lines:

6 – there should be RDS host created by Terraform

12 – here we define which database we want to observe

15:18 – here you can find deeper explanation about those lines: https://debezium.io/blog/2017/09/25/streaming-to-another-database/

Nice explanation about parameters: https://debezium.io/docs/connectors/mysql/#example-configuration

Save that json to source.json file and then you can execute following command:

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json"  -d @source.json http://$public_node_ip:30001/connectors/

Where public_node_ip is Eks node. That curl will configure connector so stream of changes in database should start.

Load data

We need to put some data in AWS MySQL which can be streamed now to Kafka. To do that we can run following sql script:


 CREATE TABLE IF NOT EXISTS my_debezium_test (
  ID int NOT NULL,
  test varchar(255),
  PRIMARY KEY (ID)
);

INSERT INTO my_debezium_test VALUES(1, 'hello!');
INSERT INTO my_debezium_test VALUES(2, 'world!');
INSERT INTO my_debezium_test VALUES(3, 'it rox!!');

UPDATE my_debezium_test SET test = "my friend!" WHERE ID = 2;

DELETE FROM my_debezium_test WHERE ID = 1;

So we create my_debezium_test table with ID field as PK and then make some inserts, update and delete to observe what will happen. Debezium should capture those changes and stream them to Kafka in real time.

Tip: you can use docker to execute mysql client to run that SQL:

sql=$(cat rds.sql)
docker run -it  --rm mysql bash -c "echo '$sql' | mysql -h${rds_endpoint_without_port} -umymysql -pmysqlmysql mydb"

Start on-prem deployment

To make quick on-prem deployment we can use docker-compose where we need to start Kafka connector which will connect to Kafka in AWS and consume changes stream, pushing them to PostgreSQL.

version: '3'

volumes:
  postgres-data:

services:
  debezium:
    image: debezium/connect-jdbc:0.8
    build:
      context: .
    ports:
      - 8083:8083
    environment:
      - BOOTSTRAP_SERVERS=${PUBLIC_NODE_IP}:30002
      - GROUP_ID=debezium-connect-sink
      - CONFIG_STORAGE_TOPIC=debezium-connect-sink_config
      - OFFSET_STORAGE_TOPIC=debezium-connect-sink_offset

  postgres:
    image: postgres
    volumes:
      - postgres-data:/var/lib/postgresql/data

Important notes to lines:

4 and 22 – we define docker named volume to store PostgreSQL data in it

13 – again we need to put connection to Kafka using EKS node IP and also place configuration required by Kafka Connect interface

10 – that line says that we need to build custom connect, let’s take a look on that

FROM debezium/connect:0.8
ENV KAFKA_CONNECT_JDBC_DIR=$KAFKA_CONNECT_PLUGINS_DIR/kafka-connect-jdbc

# Deploy PostgreSQL JDBC Driver
RUN cd /kafka/libs && curl -sO https://jdbc.postgresql.org/download/postgresql-42.1.4.jar

# Deploy Kafka Connect JDBC
RUN mkdir $KAFKA_CONNECT_JDBC_DIR && cd $KAFKA_CONNECT_JDBC_DIR &&\
	curl -sO http://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/3.3.0/kafka-connect-jdbc-3.3.0.jar

So we need to extend debezium/connect image to add JDBC driver specific for PostgreSQL so our connector can consume changes from Kafka and push them to PostgreSQL.

So next step is to start those containers and configure JDBC sink using following JSON config:

 {
  "name": "jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "tasks.max": "1",
    "topics": "my_debezium_test",
    "connection.url": "jdbc:postgresql://postgres:5432/postgres?user=postgres&password=postgres",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
    "auto.create": "true",
    "insert.mode": "upsert",
    "pk.fields": "ID",
    "pk.mode": "record_value"
  }
}

Important notes to lines:

6 – it’s name of table which will be consumed, created in AWS MySQL in previous step

8,9 – explanation here https://debezium.io/blog/2017/09/25/streaming-to-another-database/

12 – Primary Key field name, created in AWS MySQL in previous step

After that configuration captured changes should be pushed to on-prem PostgreSQL database.

Results

So our PostgreSQL should contain now automatically created my_debezium_test table but what about data? There is small surprise here. INSERT and UPDATE are perfectly replicated but there is problem with DELETE operation as it’s currently dropped by Jdbc sink connector. There is opened pull request for that on Github https://github.com/confluentinc/kafka-connect-jdbc/pull/282

I asked about this on my Twitter and I received response from Gunnar Morling that workaround for that problem is coming in Debezium 0.9.x version:

Edit:

Debezium guys decided to deliver that feature faster than expected! Nice!

 

So as conclusion I can say that solution is not perfect and in case of pure database replication there are some missing points but in my opinion it’s very big step forward as replicating database from MySQL to PostgreSQL in producer/consumer pattern via queue is something very innovative and amazing – that define new way to create reliable and HA global applications distributed across whole world.

What stores Kubernetes in Etcd?

Introduction

Kubernetes uses Etcd to store information what’s happening on cluster – it means that master nodes read/write data from/to Etcd cluster to maintain cluster state. Etcd itself is simple distributed key-value store which uses Raft consensus algorithm.

As I like to dig deeper and I couldn’t find how looks data structure stored in Etcd I decided to make a little reverse engineering on my local Minikube cluster. To go trough that article by yourself you need installed Minikube on your machine.

Kubernetes version used: 1.10

How to get into Etcd in Minikube

Firstly we need to discover way how to see data stored in Etcd in Minikube – it’s not so trivial as Etcd has some security mechanisms which blocks us to just read data from any container on Kubernetes or local machine. That security is natural as Kubernetes store there e.g. secrets, private keys, etc. as plain text if encryption is not activated (default). Also writing data directly into Etcd gives us effectively admin rights on cluster.

Firtly start your minikube:

minikube start

And launch Kubernetes dashboard:

minikube dashboard

In kube-system namespace we can find etcd-minikube POD which in we are interested – in start command section we can see some useful things:

etcd
--advertise-client-urls=https://127.0.0.1:2379
--cert-file=/var/lib/localkube/certs/etcd/server.crt
--trusted-ca-file=/var/lib/localkube/certs/etcd/ca.crt
--peer-cert-file=/var/lib/localkube/certs/etcd/peer.crt
--peer-key-file=/var/lib/localkube/certs/etcd/peer.key
--listen-client-urls=https://127.0.0.1:2379
--client-cert-auth=true
--peer-client-cert-auth=true
--data-dir=/data/minikube
--key-file=/var/lib/localkube/certs/etcd/server.key
--peer-trusted-ca-file=/var/lib/localkube/certs/etcd/ca.crt
  1. We see that client interface is bound to 127.0.0.1 so we are able to access Etcd only using localhost (not for example POD IP)
  2. We see that Etcd listen on 2379 port what is default Etcd port
  3. –client-cert-auth=true says us that certificate authentication is activated what means that we need trusted client certificate to get into Etcd

That’s a little strange – how any Kubernetes service can access Etcd if it’s listening only on localhost? So answer is in POD descriptor:

"hostNetwork": true

What means that Etcd’s docker container bind to host’s network interface. So who is connecting on localhost:2379 on Minikube? Let’s go into Minikube VM by using:

minikube ssh

In Minikube VM we have Sysdig available so we can check who is using that port:

sudo sysdig fd.port=2379

....
194074 10:30:39.093997519 1 etcd (3379) > read fd=20(<4t>127.0.0.1:57152->127.0.0.1:2379) size=1024 
194075 10:30:39.094001352 1 etcd (3379) < read res=-11(EAGAIN) data= 
194219 10:30:39.095750456 0 etcd (3236) > write fd=20(<4t>127.0.0.1:57152->127.0.0.1:2379) size=46 
194229 10:30:39.095821582 0 etcd (3236) < write res=46 data=....)......#..+jk.....I5o..N{z....e.9.;..wf... 
194240 10:30:39.095941041 1 kube-apiserver (3360) > read fd=16(<4t>127.0.0.1:57152->127.0.0.1:2379) size=2048 
194244 10:30:39.095983703 1 kube-apiserver (3360) < read res=46 data=....)......#..+jk.....I5o..N{z....e.9.;..wf... 
194247 10:30:39.096011985 1 kube-apiserver (3360) > read fd=16(<4t>127.0.0.1:57152->127.0.0.1:2379) size=2048 
194249 10:30:39.096014883 1 kube-apiserver (3360) < read res=-11(EAGAIN) data= 
.....

We see that kube-apiserver (as expected) communicate with Etcd over localhost (so kube-apiserver is also bound to host’s network interface). So let’s take a look on kubeapi starting command in Dashboard:

kube-apiserver
--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
--enable-bootstrap-token-auth=true
--requestheader-extra-headers-prefix=X-Remote-Extra-
...
...
--etcd-servers=https://127.0.0.1:2379
--etcd-cafile=/var/lib/localkube/certs/etcd/ca.crt
--etcd-certfile=/var/lib/localkube/certs/apiserver-etcd-client.crt
--etcd-keyfile=/var/lib/localkube/certs/apiserver-etcd-client.key

What is important here:

  1. We have confirmation that etcd and kube-apiserver communicate over localhost
  2. CA file is required to accept Etcd server certificate
  3. Kube-apiserver contains own certificates to authenticate in Etcd

To see data stored in Etcd we can borrow certificates from kube-apiserver and inject them into Etcd container as there is installed etcdctl – CLI which allows to access data in Etcd.

We need to find etcd and apiserver pods. Kubectl is automatically configured to target minikube:

kubectl get pods --namespace kube-system                                                                                              
NAME                                    READY     STATUS    RESTARTS   AGE
etcd-minikube                           1/1       Running   0          3m
kube-addon-manager-minikube             1/1       Running   2          4d
kube-apiserver-minikube                 1/1       Running   0          3m
kube-controller-manager-minikube        1/1       Running   0          3m
kube-dns-86f4d74b45-hpzx9               3/3       Running   9          4d
kube-proxy-tzlzs                        1/1       Running   0          2m
kube-scheduler-minikube                 1/1       Running   2          4d
kubernetes-dashboard-5498ccf677-fs5vk   1/1       Running   6          4d
storage-provisioner                     1/1       Running   6          4d

So we see etcd and apiserver pods – we need to copy certificates from apiserver to etcd:

kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/localkube/certs/apiserver-etcd-client.crt apiserver-etcd-client.crt
kubectl cp --namespace kube-system apiserver-etcd-client.crt etcd-minikube:var/lib/localkube/certs/

kubectl cp --namespace kube-system kube-apiserver-minikube:var/lib/localkube/certs/apiserver-etcd-client.key apiserver-etcd-client.key
kubectl cp --namespace kube-system apiserver-etcd-client.key etcd-minikube:var/lib/localkube/certs/

And go into Etcd container:

kubectl exec -it --namespace kube-system etcd-minikube sh

We need to set etcdctl tool to v3 API version using following environment variable:

export ETCDCTL_API=3

And test if we have access to Etcd:

/var/lib/localkube/certs # etcdctl --cacert="etcd/ca.crt" --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt endpoint status 
127.0.0.1:2379, 8e9e05c52164694d, 3.1.12, 2.2 MB, true, 3, 18349

Data structure description

To get all keys from Etcd we need to type:

/var/lib/localkube/certs # etcdctl --cacert="etcd/ca.crt" --key=apiserver-etcd-client.key --cert=apiserver-etcd-client.crt get / --prefix --keys-only

Output is really big and divided into some namespaces – let’s look on them.

 

/registry/apiregistration.k8s.io/apiservices/{version}.{api name}

Contains definition of API Services in Kubernetes so we can find there all existing core APIs used by Kubernetes like /registry/apiregistration.k8s.io/apiservices/v1.batch or /registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io or custom APIs definition (see https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/aggregation.md). You can get information about APIs by reading value of that key in Etcd (you will get human-readable json) or in more friendly way by using kubectl get apiservice v1beta1.authorization.k8s.io -o json (the same value as in direct Etcd access)

/registry/clusterroles/{role name}

Contains definition of all cluster-wide roles in Kubernetes so we can find there things like /registry/clusterroles/cluster-admin or /registry/clusterroles/system:kube-scheduler. Data in Etcd is human readable but hard to understand – we can see there some actions like get, patch, update on some parts of API

/registry/clusterrolebindings/{entity name}

Contains binding between roles and users/groups/service accounts which are cluster-wide so we can find there things like /registry/clusterrolebindings/cluster-admin or /registry/clusterrolebindings/kubeadm:kubelet-bootstrap. Data in Etcd is human readable but hard to understand.

/registry/roles/{namespace}/{role name} and /registry/rolebindings/{namespace}/{entity name}

Same story as in cluster roles/bindings but scoped by namespace e.g. /registry/roles/kube-system/system:controller:token-cleaner

/registry/serviceaccounts/{namespace}/{name}

Definition of all service accounts

/registry/configmaps/{namespace}/{map name}

All configs maps stored as yamls

/registry/controllerrevisions/{namespace}/{pod}

I found ControllerRevision resource is used to provide rollback possilibities in DaemonSet and StatefulSet (https://kubernetes.io/docs/tasks/manage-daemon/rollback-daemon-set/). In Etcd we can find snapshot of pods spec.

/registry/daemonsets/{namespace}/{name} and /registry/deployments/{namespace}/{name} etc.

Under those keys Kubernetes stores information about different deployments like DaemonSet, Deployment, ReplicaSet, Job, etc. What’s interesting in case of deployment we see there last-applied-configuration described there https://kubernetes.io/docs/concepts/overview/object-management-kubectl/declarative-config/#merge-patch-calculation

/registry/minions/{node name}

Kubernetes nodes were previously called “minions” so in Etcd name is still not changed. We see there big amount of data describing node like:

  • CPU cores
  • Memory size
  • Status of kubelet: e.g. kubelet has sufficient disk space available or kubelet has sufficient PID available
  • Ip address
  • Hostname
  • Docker version
  • Docker image/registry/ranges/servicenodeportss available on node

/registry/namespaces/{namespace}

Just defining namespace. There is also state of particular namespace like Active or Terminating.

/registry/pods/{namespace}/{pod name}

State of every pod running in cluster. Contains a lot of information like pod IP, mounted volumes, docker image etc.

/registry/ranges/serviceips

CIDR for services

/registry/ranges/servicenodeports

Ports range for exposing services

/registry/secrets/{namespace}/{pod}

All secrets in cluster stored as plain text in default mode. For encryption see https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

/registry/services/endpoints/{namespace}/{name}

Services definition. Kubernetes calculates which pods are selected by particular service and stores that information in service value so we can see pods ip addresses and names there.