Docker development and Kubernetes deployments post

Docker development and Kubernetes deployments

Stumbling down the path to enlightenment

View Full Source on GitLab

Why

I had a web project I was running for a while, doing local development using Docker Compose to emulate the target environment, which was actually deployed via SSH.I started to think about the disparity between environments and decided to bridge the gap with a 'prod' docker image and compose file.

This initially seemed like an easy win as I'd been using compose for local development for a while already, but instead I ended up in a wormhole of internet opinions and after some consideration, decided that Kubernetes is the better approach.

I have mostly hobbyist experience with Kubernetes via homelab tools like Rancher, but took this as a chance to get more hands on experience.

Translating Current Workflow

Docker Compose has been great for me in local development, and was an accurate representation of my target environment, so I looked into options for porting and found the conversion tool Kompose

Komposes allows conversion of docker compose files into Kubernetes deployment files, which ticked the box for me and let me move on to experimenting.

There were few kompose specific changes I had to make, like the LoadBalancer label to mark a service as an ingress service, allowing it to be accessed externally.

Kubernetes As a Service

I have most of my Cloud experience with Azure, so decided it's the best place to start and picked Azure Kubernetes Service (AKS) as a target.

The primary goal here is to then create an instance of AKS and store the access keys for kubectl use in the pipeline.

GitLab offers a solution for this, if you configure cluster access within the Project, an automatic KUBECONFIG variable is generated. The variable contains the path to a valid kubeconfig file which can then be used with kubectl to access the cluster.

I had issues with this, as the now deprecated certificate access didn't work for me. I decided to take a more familiar route, and made use of Terraform to provision the AKS, with an output variable for the raw kubeconfig.

After some fiddling, I had a fully source controlled AKS instance without having to hardcode anything silly. This replaces the iffy cluster integration requirement with a simple requirement of four Azure Resource Manager variables for Terraform authentication.

Deployment

Everything is tied together with a GitLab pipeline.

First, the pipeline builds images for the required services and pushes them to the internal GitLab Container Registry.

Second, the images are pulled and tests can be ran against them

Then, since testing has passed the images are pulled once more and tagged for release

Finally, the pipeline executes the Terraform plan and uses the resulting kubernetes instance to host the kubernetes workload converted from the docker compose files.

Payoffs and Losses

I've set up a CNAME pointing to the (current) external IP. k8s-sample.andrewkidd.co.uk

In summary, it's been fun to work with but probably overkill for most of my use cases.

As with most 'new' stacks I spent a lot of time stuck on small things that were in the end frustratingly simple.

The pipeline I originally implemented this on has increased by about 1.5x in duration due to the changes made, but gives a far more reliable, scalable and secure end result.

The Kubernetes instances I host at home are small-time single node clusters running basic docker images helped greatly by nice interfaces like Portainer, so it's nice to get a bit more involved.

After this experience I've been looking at creating a GitOps Pi Cluster to play with.

Next Steps / Considerations

This started out as a primarily Docker heavy experience but quickly became more about Kubernetes. In that transition the original goal of 1:1 deployments has been missed in exchange for ease-of-use and scalability.

  • The solution for this would be to move tooling to something like Minikube for a local k8s instance, and a Helm Chart for deployment configuration.

  • GitLab also offers Kubernetes integration, but relies on an agent being deployed in cluster. Ideally this would be part of the CI/CD pipeline for auto-config.

  • The Compose specification is pretty wild, Docker itself pretty much disregards the version and supports mix and match of version features allowing you to easily create wildly unsupported configuration files.

  • This is a barebones implementation which could be extended to include extra configuration like DNS.

Categories: devops

Tags: docker, development, deployment, devops, gitlab, kubernetes, k8s, terraform