Vault Helm Kubernetes

Vault Setup in Kubernetes

With Helm

Posted by ChenRiang on November 29, 2020

Secret management is always a serious issue in any kind of application. Although, Kubernetes has its own Secret feature could help to manage secret, but after some reading I realized that it basically encode the secret with Base64.

So, I looked into HashiCorp Vault.

HashiCorp Vault

Vault

It is an open source secret management tool that used to manage access to sensitive credential.

Refer here for more information.

Vault Setup

HashiCorp have provided a ready Helm Chart that we can use.

  1. Create the following files in our demo helm chart directory :

    • Chart.yaml

      1
2
3
4
5

       apiVersion: v1
 name: helm-vault
 version: 0.1.0
 appVersion: 1.0
 description: Helm Vault

    • requirements.yaml

      1
2
3
4
5
6

         dependencies:
   - name: vault
     version: 0.8.0
     repository: "https://helm.releases.hashicorp.com"
     condition: vault.enabled
     alias: vault

    • values.yaml
      ** We will come back to the value later.

Dev Mode

Vault provide an option to start Vault server in dev mode which does not require any further setup.

Note: Never, ever, ever run a “dev” mode server in production

In this section we will setup Vault in dev mode and validate it by running a simple KV store and read action via API.

  1. Edit values.yaml with the following value. 
    1
2
3
4
5
6

     vault:
   server:
     dev:
       enabled: true
     readinessProbe:
       enabled: false
    • Set the dev.enabled to true.
    • Set the readinessProbe.enabled to false as the probe check will query the status with https which the Vault server currently does not support HTTPS scheme

  2. Helm install the Vault.
    helm install helm-vault

  3. Once the pod is ready, we will need to port forward the vault service to localhost to play around with Vault.
    kubectl port-forward helm-vault 8200:8200

  4. Execute following command to store a KV into vault.

    1
2
3
4
5
6
7
8
9
10
11

    curl --location --request POST 'http://localhost:8200/v1/secret/data/my-secret' \
--header 'X-Vault-Token:root' \
--header 'Content-Type: application/json' \
--data '{
  "options": {
    "cas": 0
  },
  "data": {
    "secretKey": "secretValue"
  }
}'
    • We will store the KV at my-secret.
    • A token root (Dev Mode default token) is set at the header.

  5. Execute the following command to read the KV from the path we set in step4.

    1
2

    curl --location --request GET http://localhost:8200/v1/secret/data/my-secret?version=1 \
--header "X-Vault-Token: root"

    Sample KV Read response:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

    {
    "request_id": "c7305d4f-29b3-e097-a837-1149cf95e3c5",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "data": {
            "foo": "bar",
            "zip": "zap"
        },
        "metadata": {
            "created_time": "2020-11-29T04:19:17.5690113Z",
            "deletion_time": "",
            "destroyed": false,
            "version": 1
        }
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Standard mode

In this section, we will setup the vault with TSL.

SSL Certificate and Key

We will use cfssl to generate SSL certificate and key for our vault to use.

  1. Create a directory named ssl, and download the required software.

    1
2
3
4
5
6
7
8
9

     mkdir ssl
 cd ssl
   
 curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o cfssl
 chmod +x cfssl
 curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o cfssljson
 chmod +x cfssljson
 curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl-certinfo_1.4.1_linux_amd64 -o cfssl-certinfo
 chmod +x cfssl-certinfo

  2. Create CA CSR, Configuration, Vualt Certificate CSR.

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

     echo '''
 {
   "hosts": [
     "cluster.local"
   ],
   "key": {
     "algo": "rsa",
     "size": 2048
   },
   "names": [
     {
       "C": "MY",
       "L": "KualaLumpur",
       "O": "Example",
       "OU": "CA",
       "ST": "Example"
     }
   ]
 }
 '''> ca-csr.json
    
 echo '''
    
 {
   "signing": {
     "default": {
       "expiry": "8760h"
     },
     "profiles": {
       "default": {
         "usages": ["signing", "key encipherment", "server auth", "client auth"],
         "expiry": "8760h"
       }
     }
   }
 }
 '''> ca-config.json
    
    
 echo '''
 {
   "CN": "helm-vault",
   "key": {
     "algo": "rsa",
     "size": 2048
   },
   "names": [
     {
       "C": "MY",
       "L": "KualaLumpur",
       "O": "Kubernetes",
       "OU": "Vault",
       "ST": ""
     }
   ]
 }
 '''> vault-csr.json

  3. Create CA cert and key.

    1

    cfssl gencert -initca ca-csr.json | cfssljson -bare ca

  4. Generate cert and private key for Vault.

    1
2
3
4
5
6
7

    cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname="helm-vault,helm-vault.playground.svc.cluster.local,helm-vault.playground.svc,localhost,127.0.0.1" \
  -profile=default \
  vault-csr.json | cfssljson -bare helm-vault

    Note: -hostname need to follow according the kubernetes service name and namespace.
    In my case,

    • kubernetes service name = helm-vault
    • kubernetes namespace = playground
  5. At this point, you should have the below 6 files created :
    • ca.pem
    • ca-key.pem
    • ca.csr
    • helm-vault.pem
    • helm-vault-key.pem
    • helm-vault.csr

  6. Remove cfssl software.

    1

    rm cfssl cfssl-certinfo cfssljson

Configure Helm

  1. Create a directory named templates.

  2. In directory templates, create a kubernetes secret, vault-secret.yaml :

    1
2
3
4
5
6
7
8
9
10
11
12
13

        
apiVersion: v1
kind: Secret
metadata:
  name: vault-secret
data:
  helm-vault.pem: |-
    {{ .Files.Get "ssl/helm-vault.pem" | b64enc }}
  helm-vault-key.pem: |-
    {{ .Files.Get "ssl/helm-vault-key.pem" | b64enc }}
  ca.pem: |-
    {{ .Files.Get "ssl/ca.pem" | b64enc }}

  3. Configure values.yaml as below:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

    vault:
  global:
    tlsDisable: false
  server:
    dev:
      enabled: false
    standalone:
      config: |
        ui = true
    
        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-secret/helm-vault.pem"
          tls_key_file  = "/vault/userconfig/vault-secret/helm-vault-key.pem"
        }
        storage "file" {
          path = "/vault/data"
        }
    extraEnvironmentVars:
      VAULT_CACERT: /vault/userconfig/vault-secret/ca.pem
    extraVolumes:
    - type: secret
      name: vault-secret

Run Vault

  1. Helm install the Vault.
    helm install helm-vault .

  2. Unseal vault with following command
    kubectl exec pod/helm-vault-0 -- vault operator init -key-shares=1 -key-threshold=1
    You will be getting an unseal key and token by executing the command. 
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

    Unseal Key 1: 1+Pwsc6ZonA3nK51UVrB4rDqkMtTqtxyPbHT4StGkPA=
    
Initial Root Token: s.0WGR21db8KpPuBL69zowkLjl
    
Vault initialized with 1 key shares and a key threshold of 1. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 1 of these keys to unseal it
before it can start servicing requests.
    
Vault does not store the generated master key. Without at least 1 key to
reconstruct the master key, Vault will remain permanently sealed!
    
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
    • Unseal Key : 1+Pwsc6ZonA3nK51UVrB4rDqkMtTqtxyPbHT4StGkPA=
    • Root Token : s.0WGR21db8KpPuBL69zowkLjl

  3. Unseal Vault with the key obtained. 
    1

    kubectl exec pod/helm-vault-0 -- vault operator unseal 1+Pwsc6ZonA3nK51UVrB4rDqkMtTqtxyPbHT4StGkPA=


  4. Execute the command below to enable KV store feature.

    1

    kubectl exec pod/helm-vault-0 -- sh -c 'export VAULT_TOKEN="s.0WGR21db8KpPuBL69zowkLjl"  && vault secrets enable -version=1 -path secret kv'


  5. Vault is ready now.

  6. Execute following command to store a KV into vault.

    1
2
3
4
5
6
7
8
9
10
11
12
13

    curl --location --request POST 'https://localhost:8200/v1/secret/data/my-secret' \
--cacert helm-vault.pem \
--header 'X-Vault-Token:s.0WGR21db8KpPuBL69zowkLjl' \
--header 'Content-Type: application/json' \
--data '{
  "options": {
    "cas": 0
  },
  "data": {
    "secretKey": "secretValue"
  }
}'
    • We will store the KV at my-secret.
    • A token s.0WGR21db8KpPuBL69zowkLjl is set at the header.
    • helm-vault.pem is used for TSL.

  7. Execute the following command to read the KV from the path we set in step6.

    1
2
3

    curl --location --request GET https://localhost:8200/v1/secret/data/my-secret?version=1 \
--cacert helm-vault.pem \
--header "X-Vault-Token: s.0WGR21db8KpPuBL69zowkLjl"

    Sample KV Read response:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    {
    "request_id": "c9af2608-ced3-492d-f1a8-e41b2d121709",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 2764800,
    "data": {
        "data": {
            "secretKey": "secretValue"
        },
        "options": {
            "cas": 0
        }
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Reference

Vault KV API : Documentation

SSL Certificate Generation: Ref1 | Ref2

CATALOG
    FEATURED TAGS
    Elasticsearch Java Kubernetes Python Spring Azure Concurrency Druid Performance Postgres Redis Spark ZSH Blob DeltaLake Helm Vault WSL XML