Rolling Updates and Rollbacks

Rollout and Versioning:

When you create a first deployment, it triggers a rollout. A new rollout creates a new deployment revision. Whenever an application is upgraded, a new deployment revision gets created. This helps us to track the changes made to our deployment. This enables us to roll back to the previous version of deployment if necessary.

# you can check the status rollout of deployement
kubectl rollout status <deployment-name>

# to see the history of rollout
kubectl rollout history <deployment-name>

2 Types of deployment strategies:

Recreate Strategy: Destroy all the applications and recreate the newer version of the applications. During these processes applications are inaccessible to the users.
Rolling updates: This is a default deployment strategy. We don't bring down all the applications in one go instead we take down the older version and bring up the newer version one by one. This way application never goes down and the upgrade is seamless.

Under the hood, a new replica set is created for hosting a new version of the application. It deletes one instance from replicaset1 and creates a new instance in replicaset2. It repeats this process until the desired state is reached.

Suppose after upgradation you realize there is something that is not working and you want to roll back to older version of deployment.

# rollback to previou version of deployment
kubectl rollout undo <deployment-name>

This will delete new replicaset 2 and bring the older ones up back in the replicaset1. And our application is back to its older version.

Configure Applications

It means configuring commands and arguments on applications, environment variables, and secrets.

Set Environment variable in Kubernetes

ENV Property you can directly set environment variable in pod definition yaml file using env property. ENV is an array so every time starts with a dash under it.

 apiVersion: v1
 kind: Pod
 metadata:
   name: envar-demo
   labels:
     purpose: demonstrate-envars
 spec:
   containers:
   - name: envar-demo-container
     image: gcr.io/google-samples/node-hello:1.0
     # env property-like array; 
     env:
     - name: DEMO_GREETING
       value: "Hello from the environment"
     - name: DEMO_FAREWELL
       value: "Directly set env variables to pod defination yaml file"

ConfigMaps:

When you have lots of pod definition files, it gets difficult to manage environment variables. So take these variables out of the file and manage centrally using ConfigMaps. So when the pod is created, the config map gets injected to make its key-value pairs available for the application to use it.

Imperative approach:

2.1 create configmap using the command line:

kubectl create configmap \
    <configmap_name> --from-literal=<key1>=<value1> \
                  --from-literal=<key2>=<value2> \
                  --from-literal=<key3>=<value3>

2.2 create configmap file:

kubectl create configmap
    <configmap_name> --from-file=<path-to-file>

example: app-config.properties file is created with key-value pair and then create configmap using the command line:

# app-config.properties
APP_NAME: front-end
APP_COLOR: blue
APP_STAGE: prod

kubectl create configmap
    app-config --from-file=app-config.properties

Declarative approach: create a config-map.yaml file with ConfigMap kind.

apiVersion: v1
kind: ConfigMap
metadata:
  name: game-demo
data:
  # property-like keys; each key maps to a simple value
  player_initial_lives: "3"
  ui_properties_file_name: "user-interface.properties"
  # file-like keys
  game.properties: 
    enemy.types=aliens,monsters
    player.maximum-lives=5    
  user-interface.properties:
    color.good=purple
    color.bad=yellow
    allow.textmode=true

You can create ConfigMap using kubectl command:

kubectl create -f config-map.yaml

You can create as many ConfigMap files according to your application like app-config, mysql-config, nginx-config etc.

Stats about ConfigMap:

# get list of configmap
kubectl get configmaps

# describe configmap
kubectl describe configmaps

Once you created the config maps, you can inject them into pod defination file under env property as a configMapKeyRef:

inject using a single environment variable:

apiVersion: v1
kind: Pod
metadata:
  name: configmap-demo-pod
spec:
  containers:
    - name: demo
      image: alpine
      command: ["sleep", "3600"]
      env:
        # Define the environment variable
        - name: PLAYER_INITIAL_LIVES # Notice that the case is different here
                                     # from the key name in the ConfigMap.
          valueFrom:
            configMapKeyRef:
              name: game-demo        # The ConfigMap this value comes from.
              key: player_initial_lives # The key to fetch.
        - name: UI_PROPERTIES_FILE_NAME
          valueFrom:
            configMapKeyRef:
              name: game-demo
              key: ui_properties_file_name

Inject using an environment variable:

apiVersion: v1
kind: Pod
metadata:
  name: configmap-demo-pod
spec:
  containers:
    - name: demo
      image: alpine
      ports: 
        - containerPort: 8080 
      envFrom:
        - configMapRef:
              name: game-demo        # This is a ConfigMap Name, this value comes from the metadata.

inject the whole data as files in a volume:

volumes:
 - name: app-config-volume
   configMap:
      name: app-config

Secrets: used to store critical information in an encoded format such as a password.

Hence, these are not encrypted so anyone can decode secrets. So configure least-privilege access to Secrets- RBAC and avoid cheching-in secrets file on SCM. For security reasons store the secrets on AWS, Vault, Azure etc Providers and enable Encryption at Rest for Secrets so they can be stored encrypted in ETCD.

For more advance, you can use Helm Secrets and HashiCorp Vault for handling sensitive data.

 # using command line 
 kubectl create secret generic \
     <secret-name> --from-literal=<key1>=<value1> \
     <secret-name> --from-literal=<key2>=<value2>

 # example:
 kubectl create secret generic \
     app-secret --from-literal=DB_USER=root \
                --from-literal=DB_PASSWORD=password

 #-------------------------------------------

 # using a properties file  
 kubectl create secret generic \
     <secret-name> --from-file=<path-to-file>

 kubectl create secret generic \
     <secret-name> --from-file=app_secret.properties

Lets create a secret file:

 apiVersion: v1
 kind: Secret
 metadata:
   name: app-secret
 data:
   DB_HOST: <encryptedText>
   DB_USER: <encryptedText>
   DB_PASSWORD: <encryptedText>

 #-----------------------------
 apiVersion: v1
 kind: Pod
 metadata:
   name: webapp
   labels: 
     name: webapp
 spec: 
   containers:
   - name: web
     image: simple-web
     ports:
       - containerPort: 8080
     envFrom: 
     # 1. Inject secrets as a envrionment variable
       - secretRef:
              name: app-secret

 #------------------------------------
 # 2. Inject a single key using secretKeyRef
 env:
   - name: PLAYER_INITIAL_LIVES
     valueFrom:
         secretKeyRef:
           name: app-secret    
           key: DB_Password

 #------------------------------------
 # 3. Inject in a volume
 volumes:
  - name: app-config-volume
    secret:
       secretName: app-secret

Some basic commands to check:


 #To encoce and decode text:
 echo -n "mySQL" | base64
 echo -n "encoded-text" | base64 --decode

 #To check secrets
 kubectl get secrets
 kubectl describe secrets
 kubectl get secret <secret-name> -o yaml

How to save secrets into etcd

Lets use etcd command line locally:

# install etcd-client 
apt-get install etcd-client

# now check etcd command line
etcdctl 

# check the pods
kubectl get pods -n kube-system

Add secrets to etcd:

all although it does not provide encryption. So Kubernetes provide an option to enable encryption at rest.

# check whether this option is enabled or not
ps -aux | grep kube-api | grep "encryption-provider-config"

# you can also check this option is set or not in configuration file
ls /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/kube-apiserver.yaml

Now create your encryption config file (encyption.yaml): to encrypt your data

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
      - pandas.awesome.bears.example
    providers:
      - aescbc:
          keys:
            - name: key1
           # comd to create base 64 encoded secret object: head -c 32 /dev/urandom | base64
              secret: <BASE 64 ENCODED SECRET>
      - identity: {}

Now on your local machine create a path for mounting this encryption.yaml file with the pod

mkdir /etc/kubernetes/enc
mv encryption.yaml /etc/kubernetes/enc

# edit this yaml file and make 3 changes 
vi /etc/kubernetes/manifests/kube-apiserver.yaml

# 1.add this line under containers's command
# - --encryption-provider-config=/etc/kubernetes/enc/encryption.yaml 
# 2. add volume mount command
# 3. add volume host path
#------------------------------------------------------------
/etc/kubernetes/manifests/kube-apiserver.yaml file:
spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --encryption-provider-config=/etc/kubernetes/enc/encryption.yaml  # add this line
    volumeMounts:
    ...
    - name: enc                           # add this line
      mountPath: /etc/kubernetes/enc      # add this line
      readonly: true                      # add this line
    ...
  volumes:
  ...
  - name: enc                             # add this line
    hostPath:                             # add this line
      path: /etc/kubernetes/enc           # add this line
      type: DirectoryOrCreate             # add this line
  ...

# Now confirm the changes 
ps -aux | grep kube-api | grep encryption-provider-config

# Ensure all secrets are encrypted
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

For more refernce check this offical documentation: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Anjali Barodia's Blog

Anjali Barodia's Blog

Application Lifecycle Management

Rolling Updates, Rollbacks, ConfigMaps, Env Variables, Secrets, Mutli container pods, Init Containers, Self Healing Applications

Rolling Updates and Rollbacks

Configure Applications

Set Environment variable in Kubernetes

How to save secrets into etcd