Kubernetes + Heketi
Let's move on to storage in Kubernetes. The most obvious way to create persistent volumes and persistent volume claims is the manual one. You pick a driver, if needed…
> Archived article from 2018, from the old nutellino.it: commands and versions may be out of date.
> Machine translation from Italian: the original is here.
Let's move on to storage in Kubernetes. The most obvious way to create persistent volumes and persistent volume claims is the manual one.
You pick a driver, if needed, and configure the persistent volumes that will then be used by the persistent volume claims.
All of this, though, makes managing storage in kubernetes clunky and slow. That's why volume management systems were created, which let you create persistent volumes on demand from the persistent volume claims being requested. This feature relies on another Kubernetes concept called Storage Classes.
In this guide we'll see how to set up Heketi, a provisioner that manages volumes based on GlusterFS. Redundancy and HA will be built into these volumes, because we'll create a topology based on mirroring across 3 nodes.
To follow this guide step by step you need:
- An existing Kubernetes cluster
- The ability to add 3 kubernetes nodes, each with an extra disk attached to the machine (in plain words, an unformatted /dev/sdb)
In my case I have 3 extra nodes that I named differently so I can tell them apart:
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
nodokube-1.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-2.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-3.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-4.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-6-g1.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-7-g2.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1
nodokube-8-g3.c.nth-passage-183120.internal Ready <none> 1d v1.8.10-rancher1- nodokube-6-g1
- nodokube-7-g2
- nodokube-8-g3
Each of these 3 nodes has an extra 30GB ssd disk attached at /dev/sdb.
What we'll do is follow the guide at https://github.com/heketi/heketi/blob/master/docs/admin/install-kubernetes.md
Checking the firewall
In my case the nodes are all ubuntu, so I don't need to open any ports in the firewall. If they're CentOs or other distributions, you need to run the iptables commands to open the gluster ports on the nodes involved:
iptables -N HEKETI
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24007 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 24008 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m tcp --dport 2222 -j ACCEPT
iptables -A HEKETI -p tcp -m state --state NEW -m multiport --dports 49152:49251 -j ACCEPT
service iptables saveAdditional prerequisites
The official guide leaves out a few essential steps, including adding to every node in the cluster the driver to connect to a gluster volume, and giving the 3 nodes that will host the glusterfs pods the ability to handle thin provisioned volumes.
So we need to run these two commands on every node:
apt-get install -y glusterfs-client
modprobe dm_thin_poolDeploy
GlusterFS daemonset
Let's create the first file, glusterfs-daemonset.json
{
"kind": "DaemonSet",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "glusterfs",
"labels": {
"glusterfs": "deployment"
},
"annotations": {
"description": "GlusterFS Daemon Set",
"tags": "glusterfs"
}
},
"spec": {
"template": {
"metadata": {
"name": "glusterfs",
"labels": {
"glusterfs-node": "daemonset"
}
},
"spec": {
"nodeSelector": {
"storagenode" : "glusterfs"
},
"hostNetwork": true,
"containers": [
{
"image": "gluster/gluster-centos:latest",
"imagePullPolicy": "Always",
"name": "glusterfs",
"volumeMounts": [
{
"name": "glusterfs-heketi",
"mountPath": "/var/lib/heketi"
},
{
"name": "glusterfs-run",
"mountPath": "/run"
},
{
"name": "glusterfs-lvm",
"mountPath": "/run/lvm"
},
{
"name": "glusterfs-etc",
"mountPath": "/etc/glusterfs"
},
{
"name": "glusterfs-logs",
"mountPath": "/var/log/glusterfs"
},
{
"name": "glusterfs-config",
"mountPath": "/var/lib/glusterd"
},
{
"name": "glusterfs-dev",
"mountPath": "/dev"
},
{
"name": "glusterfs-cgroup",
"mountPath": "/sys/fs/cgroup"
}
],
"securityContext": {
"capabilities": {},
"privileged": true
},
"readinessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 60,
"exec": {
"command": [
"/bin/bash",
"-c",
"systemctl status glusterd.service"
]
}
},
"livenessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 60,
"exec": {
"command": [
"/bin/bash",
"-c",
"systemctl status glusterd.service"
]
}
}
}
],
"volumes": [
{
"name": "glusterfs-heketi",
"hostPath": {
"path": "/var/lib/heketi"
}
},
{
"name": "glusterfs-run"
},
{
"name": "glusterfs-lvm",
"hostPath": {
"path": "/run/lvm"
}
},
{
"name": "glusterfs-etc",
"hostPath": {
"path": "/etc/glusterfs"
}
},
{
"name": "glusterfs-logs",
"hostPath": {
"path": "/var/log/glusterfs"
}
},
{
"name": "glusterfs-config",
"hostPath": {
"path": "/var/lib/glusterd"
}
},
{
"name": "glusterfs-dev",
"hostPath": {
"path": "/dev"
}
},
{
"name": "glusterfs-cgroup",
"hostPath": {
"path": "/sys/fs/cgroup"
}
}
]
}
}
}
}And run the create command
$ kubectl create -f glusterfs-daemonset.json
daemonset "glusterfs" createdAt this point the glusterfs pods aren't running yet: first we need to label the nodes with the right key. So let's label them as specified in the daemonset
"spec": {
"nodeSelector": {
"storagenode" : "glusterfs"
}$ kubectl label node nodokube-6-g1.c.nth-passage-183120.internal storagenode=glusterfs
node "nodokube-6-g1.c.nth-passage-183120.internal" labeled
$ kubectl label node nodokube-7-g2.c.nth-passage-183120.internal storagenode=glusterfs
node "nodokube-7-g2.c.nth-passage-183120.internal" labeled
$ kubectl label node nodokube-8-g3.c.nth-passage-183120.internal storagenode=glusterfs
node "nodokube-8-g3.c.nth-passage-183120.internal" labeledNow the daemonset will find the nodes with the right label and go ahead and create the pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-296z2 0/1 Running 0 36s
glusterfs-4cd5v 0/1 Running 0 32s
glusterfs-pgvgt 0/1 Running 0 40s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-296z2 1/1 Running 0 2m
glusterfs-4cd5v 1/1 Running 0 1m
glusterfs-pgvgt 1/1 Running 0 2mHeketi service account
Now we need to create a service account for heketi, so let's create the file heketi-service-account.json
{
"apiVersion": "v1",
"kind": "ServiceAccount",
"metadata": {
"name": "heketi-service-account"
}
}and then run the create command:
$ kubectl create -f heketi-service-account.json
serviceaccount "heketi-service-account" createdNow we need to make sure this service account can control the glusterfs pods. So let's create a role binding for the service account we just created
$ kubectl create clusterrolebinding heketi-gluster-admin --clusterrole=edit --serviceaccount=default:heketi-service-account
clusterrolebinding "heketi-gluster-admin" createdNext we create a secret that will hold the configuration of the Heketi instance. The configuration must contain the kubernetes executor so that the Heketi server can control the glusterfs pods.
So let's create the file heketi.json
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "My Secret"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "My Secret"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
"executor": "kubernetes",
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"kubeexec": {
"rebalance_on_expansion": true
},
"sshexec": {
"rebalance_on_expansion": true,
"keyfile": "/etc/heketi/private_key",
"fstab": "/etc/fstab",
"port": "22",
"user": "root",
"sudo": false
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false
}WARNING in this example we've disabled authentication.
Let's create this secret:
$ kubectl create secret generic heketi-config-secret --from-file=./heketi.json
secret "heketi-config-secret" createdStarting the Heketi bootstrap pod
Now we start a bootstrap pod that will take care of creating our cluster and saving the heketi configuration for the production deploy that comes next.
Let's create the file heketi-bootstrap.json
{
"kind": "List",
"apiVersion": "v1",
"items": [
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "deploy-heketi",
"labels": {
"glusterfs": "heketi-service",
"deploy-heketi": "support"
},
"annotations": {
"description": "Exposes Heketi Service"
}
},
"spec": {
"selector": {
"name": "deploy-heketi"
},
"ports": [
{
"name": "deploy-heketi",
"port": 8080,
"targetPort": 8080
}
]
}
},
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "deploy-heketi",
"labels": {
"glusterfs": "heketi-deployment",
"deploy-heketi": "deployment"
},
"annotations": {
"description": "Defines how to deploy Heketi"
}
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"name": "deploy-heketi",
"labels": {
"name": "deploy-heketi",
"glusterfs": "heketi-pod",
"deploy-heketi": "pod"
}
},
"spec": {
"serviceAccountName": "heketi-service-account",
"containers": [
{
"image": "heketi/heketi:dev",
"imagePullPolicy": "Always",
"name": "deploy-heketi",
"env": [
{
"name": "HEKETI_EXECUTOR",
"value": "kubernetes"
},
{
"name": "HEKETI_FSTAB",
"value": "/var/lib/heketi/fstab"
},
{
"name": "HEKETI_SNAPSHOT_LIMIT",
"value": "14"
},
{
"name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
"value": "y"
}
],
"ports": [
{
"containerPort": 8080
}
],
"volumeMounts": [
{
"name": "db",
"mountPath": "/var/lib/heketi"
},
{
"name": "config",
"mountPath": "/etc/heketi"
}
],
"readinessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 3,
"httpGet": {
"path": "/hello",
"port": 8080
}
},
"livenessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 30,
"httpGet": {
"path": "/hello",
"port": 8080
}
}
}
],
"volumes": [
{
"name": "db"
},
{
"name": "config",
"secret": {
"secretName": "heketi-config-secret"
}
}
]
}
}
}
}
]
}and then launch it; a service and a deployment called deploy-heketi will be created:
$ kubectl create -f heketi-bootstrap.json
service "deploy-heketi" created
deployment "deploy-heketi" createdLet's check the pod status:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-heketi-6b496b5777-9f2rp 0/1 Running 0 26s
glusterfs-296z2 1/1 Running 0 1h
glusterfs-4cd5v 1/1 Running 0 1h
glusterfs-pgvgt 1/1 Running 0 1h
kubectl get pods
NAME READY STATUS RESTARTS AGE
deploy-heketi-6b496b5777-9f2rp 1/1 Running 0 1m
glusterfs-296z2 1/1 Running 0 1h
glusterfs-4cd5v 1/1 Running 0 1h
glusterfs-pgvgt 1/1 Running 0 1hNow that the bootstrap pod is running we can set up port forwarding on our machine to use heketi-cli (download heketi-cli from https://github.com/heketi/heketi/releases/).
Grab the name of the heketi pod, in my case deploy-heketi-6b496b5777-9f2rp, and run the following command
$ kubectl port-forward deploy-heketi-6b496b5777-9f2rp :8080
Forwarding from 127.0.0.1:64622 -> 8080In my case port 64622 on my host was forwarded to the deploy-heketi pod.
Let's check that it works:
$ curl http://localhost:64622/hello
Hello from HeketiFor convenience, let's set the environment variable holding the heketi address so we don't have to type the pod address every time
$ export HEKETI_CLI_SERVER=http://localhost:64622Loading the gluster server topology
Now we need to load the topology of our nodes running the glusterfs daemonset pods, also specifying which disk on the machine will be formatted for use with the glusterfs volumes.
Let's create the file topology-load.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"nodokube-6-g1.c.nth-passage-183120.internal"
],
"storage": [
"10.142.0.8"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"nodokube-7-g2.c.nth-passage-183120.internal"
],
"storage": [
"10.142.0.9"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"nodokube-8-g3.c.nth-passage-183120.internal"
],
"storage": [
"10.142.0.7"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
}
]
}
]
}A few parameters worth noting:
- storage: the static ip of the node running the pod
- manage: the hostname of the node, in my case nodokube-6-g1.c.nth-passage-183120.internal, etc
- devices: the path of the disk added to the virtual machine
So let's load the topology into our heketi pod via heketi-cli:
$ ./heketi-cli topology load --json=topology-load.json
Creating cluster ... ID: 9ed46f67a517d5e46b9245e1214c2c03
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node nodokube-6-g1.c.nth-passage-183120.internal ... ID: 0379d0bb941a80c6ec65fed344306602
Adding device /dev/sdb ... OK
Creating node nodokube-7-g2.c.nth-passage-183120.internal ... ID: 7d271bdb173887adc09ecfc677604378
Adding device /dev/sdb ... OK
Creating node nodokube-8-g3.c.nth-passage-183120.internal ... ID: 8a547e452d0442f7e44a317e75c9fba8
Adding device /dev/sdb ... OKSaving the heketi database to a gluster volume
Here comes a rather "magical" step. We need to create a volume provisioned by deploy-heketi that will then be used by the real heketi pod we're going to use.
Let's ask heketi to create the volume for us
$ ./heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json
$ kubectl create -f heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" createdWe have to wait for the job to complete, after which we can go ahead and delete the deploy pods
Let's check the status with:
$ kubectl get jobs
NAME DESIRED SUCCESSFUL AGE
heketi-storage-copy-job 1 1 1mLet's delete the bootstrap deploy:
$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
deployment "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-6b496b5777-9f2rp" deleted
service "deploy-heketi" deleted
secret "heketi-storage-secret" deletedCreating the long-term Heketi
Let's create the final heketi deploy file, heketi-deployment.json
{
"kind": "List",
"apiVersion": "v1",
"items": [
{
"kind": "Secret",
"apiVersion": "v1",
"metadata": {
"name": "heketi-db-backup",
"labels": {
"glusterfs": "heketi-db",
"heketi": "db"
}
},
"data": {
},
"type": "Opaque"
},
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-service",
"deploy-heketi": "support"
},
"annotations": {
"description": "Exposes Heketi Service"
}
},
"spec": {
"selector": {
"name": "heketi"
},
"ports": [
{
"name": "heketi",
"port": 8080,
"targetPort": 8080
}
]
}
},
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "heketi",
"labels": {
"glusterfs": "heketi-deployment"
},
"annotations": {
"description": "Defines how to deploy Heketi"
}
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"name": "heketi",
"labels": {
"name": "heketi",
"glusterfs": "heketi-pod"
}
},
"spec": {
"serviceAccountName": "heketi-service-account",
"containers": [
{
"image": "heketi/heketi:dev",
"imagePullPolicy": "Always",
"name": "heketi",
"env": [
{
"name": "HEKETI_EXECUTOR",
"value": "kubernetes"
},
{
"name": "HEKETI_FSTAB",
"value": "/var/lib/heketi/fstab"
},
{
"name": "HEKETI_SNAPSHOT_LIMIT",
"value": "14"
},
{
"name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
"value": "y"
}
],
"ports": [
{
"containerPort": 8080
}
],
"volumeMounts": [
{
"mountPath": "/backupdb",
"name": "heketi-db-secret"
},
{
"name": "db",
"mountPath": "/var/lib/heketi"
},
{
"name": "config",
"mountPath": "/etc/heketi"
}
],
"readinessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 3,
"httpGet": {
"path": "/hello",
"port": 8080
}
},
"livenessProbe": {
"timeoutSeconds": 3,
"initialDelaySeconds": 30,
"httpGet": {
"path": "/hello",
"port": 8080
}
}
}
],
"volumes": [
{
"name": "db",
"glusterfs": {
"endpoints": "heketi-storage-endpoints",
"path": "heketidbstorage"
}
},
{
"name": "heketi-db-secret",
"secret": {
"secretName": "heketi-db-backup"
}
},
{
"name": "config",
"secret": {
"secretName": "heketi-config-secret"
}
}
]
}
}
}
}
]
}Let's run the deploy:
$ kubectl create -f heketi-deployment.json
service "heketi" created
deployment "heketi" createdLet's set up the forward rule again to test that heketi works
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
glusterfs-296z2 1/1 Running 0 2h
glusterfs-4cd5v 1/1 Running 0 2h
glusterfs-pgvgt 1/1 Running 0 2h
heketi-84684b87c-hqgld 1/1 Running 0 24s
$ kubectl port-forward heketi-84684b87c-hqgld :8080
Forwarding from 127.0.0.1:49337 -> 8080
$ export HEKETI_CLI_SERVER=http://localhost:49337Let's check the cluster:
$ ./heketi-cli cluster list
Clusters:
Id:9ed46f67a517d5e46b9245e1214c2c03 [file][block]
$ ./heketi-cli volume list
Id:e38d822489a5a5401f30c41d3c24fb8c Cluster:9ed46f67a517d5e46b9245e1214c2c03 Name:heketidbstorageLet's create a Storage Classes
First of all we need to find out the clusterip of the heketi pod:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
heketi ClusterIP 10.43.105.131 <none> 8080/TCP 9m
heketi-storage-endpoints ClusterIP 10.43.124.82 <none> 1/TCP 14m
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 1dIn my case 10.43.105.131. Let's create the file heketi-storageclasses.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.43.105.131:8080"
restuser: "joe" # non servono nell'esempio non essendoci autenticazione
restuserkey: "My Secret Life" # non servono nell'esempio non essendoci autenticazioneAnd create it with:
$ kubectl create -f heketi-storageclasses.yaml
storageclass "gluster-heketi" createdLet's make it the default for all upcoming deployments that use a persistent volume claim:
$ kubectl patch storageclass gluster-heketi -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass "gluster-heketi" patchedTest PVC!
Finally, let's create our test persistent volume claim, and call it test-pvc.yml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster1
annotations:
volume.beta.kubernetes.io/storage-class: gluster-heketi
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2GiNow comes the moment of truth: asking heketi to create a persistent volume on the fly.
$ kubectl create -f test-pvc.yaml
persistentvolumeclaim "gluster1" createdLet's check:
$ kubectl get pvc,pv
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc/gluster1 Bound pvc-9fd5feda-3725-11e8-902a-02611b0ceca9 2Gi RWX gluster-heketi 25s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv/pvc-9fd5feda-3725-11e8-902a-02611b0ceca9 2Gi RWX Delete Bound default/gluster1 gluster-heketi 19sYEAH we did it! Now we'll never have to worry about creating pvs by hand inside kubernetes again!
$ cat ~/archivio/kubernetes-heketi.md
WARNING: this file is from 2018.
I wrote it, but a me from a few years ago: it may no longer reflect what I think or what I can do.
NOTE: machine translation from Italian, the original is here.
Continue? [y/N]