arduino 1 copilot 1 docker 3 editor-script 1 gcp 2 github 1 grafana 1 htmlhint 1 hugo 2 influxdb 1 jekyll 1 kubernetes 2 minikube 1 npm 2 plantuml 1 rpi 1 semantic-release 1 semver 1 skaffold 2 static-site-generator 3 telegraf 1 unity 1

Designing a microservices application using Kubernetes and Skaffold - part 1

Every now and then you get the chance to build something from scratch. A greenfield project! Endless possibilities often combined with the requirement to have something up and running as quickly as possible, and with low running costs.

WOW! A chance to use past experiences but hopefully not make the exact same mistakes made in the past...

The thought with this post and upcoming ones is to share some light on the thought process and design choices made when starting from scratch and also, in later parts, some details regarding the tech stack that is still forming.

The project at hand

This time the challenge was to setup an application consisting of a few frontends and a backend as well as making use of other tools and services that we either use via externally hosted APIs or self-host and use as part of our system.

The company I was contracted by had past experiences using Google Cloud Platform (GCP) which was a driving factor in choice of cloud provider. My goal was however to ensure the application was as platform agnostic as possible without compromising in development speed.

Why Kubernetes?

We decided pretty quickly to use Kubernetes for orchestration of containers. To go completely serverless for a bigger project would be interesting but felt too risky given the situation and we also knew that we needed to run at least parts of the system in containers.

My experiences using Kubernetes were slim to none at this point although I had some hands-on experience deploying and running containers in AWS ECS as well as utilizing docker + docker compose in various dev environments and tools.

The choice to go with Kubernetes felt straight forward. Arguably Kubernetes is the winner in the race of container orchestration. It’s built by Google and has more or less “native” support in GCP.

Why Skaffold?

Again my experiences using Kubernetes were limited at this point. I knew that I wanted to keep things as simple as possible, and not waste time managing a cluster.

I found that Google had a demo project of an ecommerce application using different microservices which was making use of Skaffold: https://github.com/GoogleCloudPlatform/microservices-demo

Cloning this project and watching it deploy nicely to a local minikube cluster simply bu using a single command was enough to get me excited.

Using the same concept I was quickly able to put together a similar setup for our services enabling them to be deployed to a local k8s cluster (Minikube) or a cluster in GCP GKE. Simply by deploying services in dev mode watching the filesystem for changes it was easy to deploy a full environment that was watching/rebuilding the corresponding service on any code change. Working primarily with Node.js where most frameworks has this built in the setup was a breeze.

Also I found it positive that Skaffold is’nt the only tool out there to serve the same purpose. There are serveral alternatives, like Garden.io and Tilt, which seem to serve a very similar purpose. In my book this means that even if Skaffold won’t survive and/or not be sufficent for our needs, there are alternatives and a migration should hopefully be simple. It also means that others see the need for introducing such a tool into their stack.

… and oh, Skaffold is free and open-source, which of course helped to seal the deal.

Monorepo vs Multirepo (and the 12-factor app guidelines)

Another question was the ever so important choice between monorepo and multirepo. Personally I have a fair amount of experience of the multi repo setup. At first it’s easy but the complexity grows exponentially with the system. Dependency management may be easy to setup but the problems start when you have a system to maintain whilst wanting to introduce new features and or make bigger structural changes. Unless you have a system of fully isolated components you will end upp with complex branching situations and build/deployment procedures to support them.

The 12-factor codebase rule says it nicely: “There is always a one-to-one correlation between the codebase and the app. If there are multiple codebases, it’s not an app – it’s a distributed system. Each component in a distributed system is an app, and each can individually comply with twelve-factor.”.

There is a line to draw here though. As I mentioned above part of this system will exist of other self-hosted “applications”/or services developed and maintained by others. In these situations we treat them separately and our Skaffold/Kubernetes setup is only depending on such services in their output (docker image) form.

Keep your secrets secure from day 1

Another fundamental part of any kind of system is how to manage it’s secrets.

There are a few basic rules to adhere:

Having previously worked with AWS I found the most effective way to handle secrets was to rely on their secret key/value service, AWS Secrets Manager. GCP has an equivalent service, GCP Secret Manager, which is pretty much 1:1 to the former. The use of such a service can easily be implemented in any environment and you can manage access to secrets depending on the IAM user authenticated, regardless if it’s human user or service user. A secret is not limited to a password but can be used to store complex data, jsons or hashes.

So put your secrets in a secure key value store like GCP Secret Manager and feed your containers deploy-time with environment variables or even have your services fetch the secrets runtime when they boot up. In the latter approach automatically rotated secrets is easier to implement.

Some outtakes from part 1

Onwards to part 2 here.


Feel free to get in touch if you have any questions.
Click to reveal contact details dude
skaffold kubernetes GCP