Gitlab Multi-tenant setup with Kubernets

Setting a CI/CD pipeline for a complex multi-tenant project it’s a lot of work! When we searched this topic we couldn’t find many helpful blogs so we decided to write our own after we crossed the swamp of the learning curve.

This setup it’s one where aside from existing a core, you may have handmade code for every different client that you don’t want in your core.

The repository setup

The basic layout we work with it’s represented by this diagram, I will explain each of the branches purposes

staging-major: All the major features come here to be staged for a new major release, after a peer-review and a automated testing battery

version-x: Whenever we have all the major features we want for a new release, we create a version-<number> branch, where we do a manual QA of all the features and we fix whatever may not be working properly. We do this in a separate branch because the QA process may take a long time (days) and we don’t want to lock staging-major during this review, so our devs can keep adding major features for the next release.

master: this is where the stable code of our setup it’s always staged

staging-minor: Here we stage all our minor features and bugfixes, so we don’t create one deployment per feature but we accumulate about a day or two worth of small features and deploy them all together.

staging-client: Specially crafted features for a particular client are staged in this branch.

master-client: The live code for a particular client

Gitlab project setup

First you want to go into your project settings under repository and protect all the branches that no one should be touching without permission

Settings -> Repository -> Protected Branches

Then we use the Settings => CI/CD variables to setup all the secrets that should not be inside your repository

And lastly, you need to setup your gitlab runner. We run this using a Google Cloud gitlab runner with docker machine, which spawn new instances for the testing process. Note: we avoid preemptive VMs as it was more of a headache than a money saving

CI/CD setup (.gitlab-ci.yml)

The first important part of this setup it’s to have your CI/CD code split in different files, as not to drive yourself crazy with a huge .gitlab-ci.yml file

.gitlab-ci.yml file

Testing

testing pipeline

We define that we only run automated testing on branches going “upstream”, meaning that a change going from staging-minor to master will get tested, but something going from master to staging-minor will not. This saves us testing time, as the CI pipeline may take a long while to run.

Configuration to avoid downstream testing

Everywhere downstream of master it’s tested against a QA database, while everything upstream of master will be test against a copy of that client’s database. This helps us minimize the risk that a particular feature or database configuration of that client may break a release without us noticing.

Deployments

Deployment pipeline

We deploy our tenants to different Kubernetes clusters, and in order to have a semblance of order we store the configurations for each client inside the repository (¡not credentials of course!). This way we have a file that only exist in each client and it’s never sent downstream.

sample configuration file that lives in staging-clientX and master-clientX

In order to use our environment variables (defined on the CI/CD configurations or on the .deploymeny-config-variables) we created a small templating script yaml_replace_envs which reemplaces the variables inside all our K8s yamls

step that prepares our K8s .ymls

Deployment .yml which is replaced by our templating

Then we trigger a backup job before the deploy, and lastly the deployment of the application and it’s monitoring tools (We use monitoring as code inside the repository)

example of our deployment step

Quality of life and restrictions

Developers make mistakes, so in order to avoid someone merging changes between two incompatible branches we added a test that runs on every Merge Request that checks the origin and target branch and aborts it in case it’s not allowed (Like staging-client going towards staging-minor)

We also automated the creation of Merge Requests, like when we merge a minor version to master, one MR it’s created for each client that goes from master to staging-client. This is paired with a python bot that auto-accept merge requests that fulfill certain conditions, for example a MR that’s going downstream it’s always auto-accepted, and a MR from master to staging-client that fulfilled the testing battery it’s also accepted

sample of our QoL steps

And that’s it! A sneak peak into our project configuration, hopefully this is helpful for you, if you have any particular questions or would like a sample of some of our scripts let me know in the comments below.