Journey to Kubernetes - PART 9000 - DigitalOcean

I want a worry free k8s!

Hi everyone!
As you may have noticed, I never finished the guides for running a kubernetes cluster on hetzner.
Partly for lack of time, partly because hetzner isn't a turnkey system.
Right now my whole infrastructure is still there, but I'm planning to move elsewhere.

That's why I've been studying DigitalOcean, which finally has mature enough services to host a kubernetes cluster.

At this link https://gitlab.com/nutellino-digitalocean-kubernetes-playground/terraform you'll find the repository for this article, with the whole recipe needed to create a kubernetes cluster and deploy the apps using just one technology: terraform

By now it's my de facto standard for managing infrastructure. Mostly because it's clear, it works well, and it's versatile. In very few steps I went from AWS to DigitalOcean simply by using terraform, without having to learn any CLI commands.

So I'll walk you through, in Italian (the readme.md explains them in English), all the steps needed to deploy your cluster!

Introduction

The cluster we're going to create will be made up of several DigitalOcean pieces:

• K8s cluster
• Managed mysql database
• Load balancer
• Volumes
• AWS DNS

As you can see I also included AWS DNS, because terraform makes it easy to use multicloud resources. I'll also use AWS S3 to store the terraform state.

Prerequisites on your machine:

• terraform (v0.11)
• kubectl command line
• helm command line
• mysql command line
• doctl command line
• An S3 Bucket on AWS S3 to store the terraform state. These states will be used as remote states and their outputs will be used as variables in the following projects

Directory structure

• 1_kubernetes: Creates a kubernetes cluster with autoscaling enabled, custom cert manager CRDs pre-installed and the prerequisites for helm.
• 2_database: Creates a mysql database on digital ocean
• 3_app-databases: Creates users and databases inside the database we just created
• 4_helm_system: deploys all the system charts required to have a fully working cluster.
• 5_dns: Project that fetches the ip of the load balancer automatically created by nginx-ingress and uses it to set two A records on AWSD
• 6_helm_apps: Helm project to deploy two complete wordpress sites inside the cluster

Creating the infrastructure

In short, to create the infrastructure we'll go into each folder in order, configure variables and backend, and run terraform apply.

Destroying the infrastructure

Go through the folders in reverse order and run terraform destroy. Check on the digital ocean web console that everything has been deleted.

Let's start the deploy

Below we'll configure and deploy each project step by step.

You'll need the aws cli configured in your current shell. See here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

You'll also need the digital ocean token. You can create it from the API menu:

[do_menu: image not recovered]

To explicitly pass the token to the terraform provider, export the variable:

export TF_VAR_digital_ocean_token="123123123123123123123123"

You can also save the line in your .bash_profile or, if you use zsh like me, .zshrc.

1 kubernetes

This project takes care of creating the kubernetes cluster.
It will also create:

• Helm namespace
• cert-manager CRDs
• doks-metrics-server (digitalocean metrics server to enable hpa autoscaling)

Configuration

Create a terraform_backend.tf file following the example:

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/kubernetes/tfstate"
    region = "eu-west-1"
  }
}

• terraform.nutellino-playground is my s3 bucket that will hold the state file
• terraform-do/kubernetes/tfstate is the resource path
• eu-west-1 is the region where the bucket lives

Then you can go ahead and configure the variables in your name.auto.tfvars file:

In my case

aws_region = "eu-west-1"
do_region = "fra1"
kubeconfig_path = "~/.kube/configdigitalocean"
namespace = "nut-auriga"

Starting the creation

So let's run the two commands:

terraform init
terraform apply

Once it's done, we can look for the cluster in our cloud console:

[after_kubernetes: image not recovered]

We'll also have the kubeconfig file saved on our machine, in my example ~/.kube/configdigitalocean

We can then activate the kubectl context with the command:

export KUBECONFIG=/Users/samuelechiocca/.kube/configdigitalocean

2 database

This project creates a managed mysql database.

There will also be some outputs (configured in outputs.tf) that will be needed in the following projects.

Configuration

Create a terraform_backend.tf file following the example:

MANDATORY: change the backend key!

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/database/tfstate"
    region = "eu-west-1"
  }
}

Then you can go ahead and configure the variables in your name.auto.tfvars file:

In my case

aws_region = "eu-west-1"
do_region = "fra1"
namespace = "nut-auriga"

Starting the creation

So let's run the two commands:

terraform init
terraform apply

Let's look for our database on digitalocean:

[after_database: image not recovered]

3 app-database

This project creates users and databases inside our mysql database. Since I'm lazy I have no intention of getting my hands dirty connecting to the database and running queries. I'd rather let terraform handle it.

Create a terraform_backend.tf file following the example:

MANDATORY: change the backend key!

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/app-databases/tfstate"
    region = "eu-west-1"
  }
}

Also create the remote_states.tf file with the references to the state of the 2_database project

data "terraform_remote_state" "database" {
  backend = "s3"
  config {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/database/tfstate"
    region = "eu-west-1"
  }
}

This example contains unencrypted passwords; in the real world you'll have to change how passwords are handled, so they don't end up committed to a version control system. One option is git-crypt.

Then you can go ahead and configure the variables in your name.auto.tfvars file:

In my case

aws_region = "eu-west-1"
namespace = "nut-auriga"
fix = "1"

fix is a parameter that, if changed in a later apply, triggers the deploy of the null_resources that switch the user's authentication to mysql_native_password.

Starting the creation

So let's run the two commands:

terraform init
terraform apply

In this example two users and two databases will be created.

We can then check that these users/dbs exist in the console:

[after_db_users: image not recovered]

4 helm_system

We can finally get to the fun part. Deploying, via Helm charts, all the subsystems k8s needs to handle ingress, volumes and ssl certificates.

So we'll deploy:

• cert-manager to automatically manage SSL certificates
• two cluster issuers using the helm chart I made specifically for this, which you can find here: https://gitlab.com/nutellino-digitalocean-kubernetes-playground/charts/tree/master/charts/cert-issuers
• nginx-ingress controller, which will automatically create a digitalocean loadbalancer
• nfs-provisioner to automatically manage NFS pvcs in the cluster (so we can have ReadWriteMany volumes

As always, create a terraform_backend.tf file following the example:

MANDATORY: change the backend key!

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/helm-system/tfstate"
    region = "eu-west-1"
  }
}

Then you can (as always) go ahead and configure the variables in your name.auto.tfvars file:

In my case

region="eu-west-1"
kubeconfig_path = "~/.kube/configdigitalocean"
email_cert_issuer = "samuele@nutellino.it"

Starting the creation

So let's run the two commands:

terraform init
terraform apply

You'll probably have to run terraform apply again, because sometimes the apiserver replies with an unable to handle request

* helm_release.nfs-provisioner: rpc error: code = Unknown desc = Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: webhook.cert-manager.io/v1beta1: the server is currently unable to handle the request

A second terraform apply will sort things out.

Let's check two things in the console:

First, we'll find the brand new load balancer:

[after_loadbalancer: image not recovered]

and second, a volume that has been attached to our nfs-provisioner

[after_nfs: image not recovered]

And with that the cluster setup is done! Now we can start deploying a couple of wordpress sites into it.

5 dns

This project fetches the ip of the load balancer we just created and uses it in the two A records we're going to create. The domain I'll use is tuttorotto.it, and I'll create two linked records:

• website1.tuttorotto.it
• website2.tuttorotto.it

As always, create a terraform_backend.tf file following the example:

MANDATORY: change the backend key!

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/dns/tfstate"
    region = "eu-west-1"
  }
}

Then you can (as always) go ahead and configure the variables in your name.auto.tfvars file:

In my case

aws_region = "eu-west-1"
domain = "tuttorotto.it"
kubeconfig_path = "~/.kube/configdigitalocean"

Starting the creation

So let's run the two commands:

terraform init
terraform apply

This way we'll have our two dns records created and ready to use. Wait a bit for propagation and then move on to the next step.

6 helm_apps

In this last project we can FINALLY deploy our two wordpress sites! :)

They'll be sites with:
* volume on a dynamic nfs pvc * database on the digitalocean mysql cluster * let's encrypt SSL certificate managed by cert-manager

As always, create a terraform_backend.tf file following the example:

MANDATORY: change the backend key!

terraform {
  backend "s3" {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/helm-apps/tfstate"
    region = "eu-west-1"
  }
}

Also create the remote_states.tf file with the references to the state of the 3_app-databases project, to fetch users, passwords, endpoints, etc.

data "terraform_remote_state" "app-databases" {
  backend = "s3"
  config {
    bucket = "terraform.nutellino-playground"
    key    = "terraform-do/app-databases/tfstate"
    region = "eu-west-1"
  }
}

Let's configure the name.auto.tfvars file:

In my case

aws_region="eu-west-1"
kubeconfig_path = "~/.kube/configdigitalocean"

Starting the creation

So let's run the two commands:

terraform init
terraform apply

This deploy will also scale our worker pool up to 2: since they're configured as very small nodes (1vcpu and 2Gb ram), the cpu requested by the two wordpress deployments is too much for a single node.

Once the deploy is complete, let's check that the two sites work:

[after_helm: image not recovered]

Conclusions

I'll probably decide to move my whole cluster to digitalocean. The resources available aren't as many as on AWS, but they're the bare minimum you need for a worry free cluster with all the k8s best practices!