We moved our workloads to Kubernetes and now want to run our tests in the cluster. In this series I describe our journey with Testkube. This setup works for us, your milage may vary. You can view all posts in this series by filtering on tag testkube-journey

GitOps

TLDR: add configurations so Kustomize understands the TestWorkflows CRD.

We use GitOps for kubernetes deployments. This means files in a git repository that can be edited from every tool that has access. On a regular interval (or by git triggers) the GitOps tool will pull the latest version from git and apply it.

Our weapon of choice is ArgoCD and the files in Git are Kustomize file(s). This means we can update the tag for our tests container from the command line.

Kustomize edit set image internal-registry-address/company/e2etests:NEW_TAG

This line actually adds or changes some lines in the kustomization.yaml file.

apiVersion: Kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- playwright-selfservice-testworkflow.yaml
- rbac.yaml

images:
- name: internal-registry-address/company/e2etests
  newTag: "NEW_TAG"

But when ArgoCD deploys this the image tag is not updated. The default implementation can update the image tag in known kubernetes resources like deployments, jobs and statefulsets. It does not know about the image for a step in TestWorkflows. This is where configurations are used.

In the stepimages.yaml below we tell Kustomize where the images must be applied when the kind is TestWorkflow.

images:
- path: spec/steps/container/image
  kind: TestWorkflow
- path: spec/steps/run/image
  kind: TestWorkflow

Add the stepimages.yaml to the configurations of the Kustomization and the image tag is updated.

apiVersion: Kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- playwright-selfservice-testworkflow.yaml
- rbac.yaml

configurations:
- stepimages.yaml

images:
- name: internal-registry-address/company/e2etests
  newTag: "NEW_TAG"

Whenever there is a new build of the e2etests image the pipeline can update the tag in our TestWorkflow. With GitOps this change is applied in kubernetes and we are using the latest version of our tests.