gitops-playground

Creates a complete GitOps-based operational stack on your Kubernetes clusters:

• Deployment: GitOps via Argo CD with a ready-to-use repo structure
• Monitoring: Prometheus and Grafana
• Secrets Management: Vault and External Secrets Operator
• Notifications/Alerts: Grafana and ArgoCD can be predefined with either an external mailserver or MailHog for demo purposes.
• Pipelines: Example applications using Jenkins with the gitops-build-lib and SCM-Manager
• Ingress Controller: ingress-nginx
• Certificate Management: (planned)
• Runs on:
  • local cluster (try it with only one command),
  • in the public cloud,
  • and even air-gapped environments (work in progress).

The gitops-playground is derived from our experiences in consulting, operating the myCloudogu platform and is used in our GitOps trainings for both Flux and ArgoCD.

TL;DR

You can try the GitOps Playground on a local Kubernetes cluster by running a single command:

bash <(curl -s \
  https://raw.githubusercontent.com/cloudogu/gitops-playground/main/scripts/init-cluster.sh) \
  && docker run --rm -t --pull=always -u $(id -u) \
    -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \
    --net=host \
    ghcr.io/cloudogu/gitops-playground --yes --argocd --ingress-nginx --base-url=http://localhost
# If you want to try all features, you might want to add these params: --mail --monitoring --vault=dev

Note that on some linux distros like debian do not support subdomains of localhost. There you might have to use --base-url=http://local.gd (see local ingresses).

See the list of applications to get started.

We recommend running this command as an unprivileged user, that is inside the docker group.

Table of contents

<!-- Update with `doctoc --notitle README.md --maxlevel 5`. See https://github.com/thlorenz/doctoc --> <!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

• What is the GitOps Playground?
• Installation
  • Overview
  • Create Cluster
  • Apply playground
    • Apply via Docker (local cluster)
    • Apply via kubectl (remote cluster)
    • Additional parameters
      • Configuration file
      • Deploy Ingress Controller
      • Deploy Ingresses
      • Deploy GitOps operators
      • Deploy with local Cloudogu Ecosystem
      • Deploy with productive Cloudogu Ecosystem and GCR
      • Override default images
      • Argo CD-Notifications
      • Monitoring
      • Mail server
      • MailHog
      • External Mailserver
      • Secrets Management
  • Remove playground
  • Running on Windows or Mac
    • Mac and Windows WSL
    • Windows Docker Desktop
• Stack
  • Credentials
  • Argo CD
    • Why not use argocd-autopilot?
    • cluster-resources
  • Jenkins
  • SCM-Manager
  • Monitoring tools
  • Secrets Management Tools
    • dev mode
    • prod mode
    • Example app
  • Example Applications
    • PetClinic with plain k8s resources
    • PetClinic with helm
    • 3rd Party app (NGINX) with helm, templated in Jenkins
    • 3rd Party app (NGINX) with helm, using Helm dependency mechanism
• Development

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

What is the GitOps Playground?

The GitOps Playground provides a reproducible environment for setting up a GitOps-Stack. It provides an image for automatically setting up a Kubernetes Cluster including CI-server (Jenkins), source code management (SCM-Manager), Monitoring and Alerting (Prometheus, Grafana, MailHog), Secrets Management (Hashicorp Vault and External Secrets Operator) and of course Argo CD as GitOps operator.

The playground also deploys a number of example applications.

The GitOps Playground lowers the barriers for operating your application on Kubernetes using GitOps. It creates a complete GitOps-based operational stack on your Kubernetes clusters. No need to read lots of books and operator docs, getting familiar with CLIs, ponder about GitOps Repository folder structures and promotion to different environments, etc.
The GitOps Playground is a pre-configured environment to see GitOps in motion, including more advanced use cases like notifications, monitoring and secret management.

In addition to creating an operational stack in production, you can run the playground locally, for learning and developing new features.

We aim to be compatible with various environments, e.g. OpenShift and in an air-gapped network. The support for these is work in progress.

Installation

There a several options for running the GitOps playground

• on a local k3d cluster Works best on Linux, but is possible on Windows and Mac.
• on a remote k8s cluster
• each with the option
  • to use an external Jenkins, SCM-Manager and registry (this can be run in production, e.g. with a Cloudogu Ecosystem) or
  • to run everything inside the cluster (for demo only)

The diagrams below show an overview of the playground's architecture and three scenarios for running the playground. For a simpler overview including all optional features such as monitoring and secrets management see intro at the very top.

Note that running Jenkins inside the cluster is meant for demo purposes only. The third graphic shows our production scenario with the Cloudogu EcoSystem (CES). Here better security and build performance is achieved using ephemeral Jenkins build agents spawned in the cloud.

Overview

Playground on local machine	Production environment with Cloudogu EcoSystem

Create Cluster

You can apply the GitOps playground to

• a local k3d cluster (see docs or script for more details):

  bash <(curl -s \
    https://raw.githubusercontent.com/cloudogu/gitops-playground/main/scripts/init-cluster.sh)

• a remote k8s cluster on Google Kubernetes Engine (e.g. via Terraform, see our docs),
• or almost any k8s cluster.
  Note that if you want to deploy Jenkins inside the cluster, you either need Docker as container runtime or set Jenkins up to run its build on an agent that provides Docker.

For the local cluster, you can avoid hitting DockerHub's rate limiting by using a mirror via the --docker-io-registry-mirror parameter.

For example:

bash <(curl -s \
    https://raw.githubusercontent.com/cloudogu/gitops-playground/main/scripts/init-cluster.sh) --docker-io-registry-mirror https://mirror.gcr.io

This parameter is passed on the containerd used by k3d.

In addition, the Jobs run by Jenkins are using the host's Docker daemon.
To avoid rate limits there, you might have to configure a mirror there as well. This can be done in the /etc/docker/daemon.json or in the config of Docker Desktop.

For example:

{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

Apply playground

You can apply the playground to your cluster using our container image ghcr.io/cloudogu/gitops-playground.
On success, the container prints a little intro on how to get started with the GitOps playground.

There are several options for running the container:

• For local k3d cluster, we recommend running the image as a local container via docker
• For remote clusters (e.g. on GKE) you can run the image inside a pod of the target cluster via kubectl.

All options offer the same parameters, see below.

Apply via Docker (local cluster)

When connecting to k3d it is easiest to apply the playground via a local container in the host network and pass k3d's kubeconfig.

CLUSTER_NAME=gitops-playground
docker pull ghcr.io/cloudogu/gitops-playground
docker run --rm -t -u $(id -u) \
  -v ~/.config/k3d/kubeconfig-${CLUSTER_NAME}.yaml:/home/.kube/config \
  --net=host \
  ghcr.io/cloudogu/gitops-playground # additional parameters go here

Note:

• docker pull in advance makes sure you have the newest image, even if you ran this command before.
  Of course, you could also specify a specific version of the image.
• Using the host network makes it possible to determine localhost and to use k3d's kubeconfig without altering, as it access the API server via a port bound to localhost.
• We run as the local user in order to avoid file permission issues with the kubeconfig-${CLUSTER_NAME}.yaml.
• If you experience issues and want to access the full log files, use the following command while the container is running:

docker exec -it \
  $(docker ps -q  --filter ancestor=ghcr.io/cloudogu/gitops-playground) \
  bash -c -- 'tail -f  -n +1 /tmp/playground-log-*'

Apply via kubectl (remote cluster)

For remote clusters it is easiest to apply the playground via kubectl. You can find info on how to install kubectl here.

# Create a temporary ServiceAccount and authorize via RBAC.
# This is needed to install CRDs, etc.
kubectl create serviceaccount gitops-playground-job-executer -n default
kubectl create clusterrolebinding gitops-playground-job-executer \
  --clusterrole=cluster-admin \
  --serviceaccount=default:gitops-playground-job-executer

# Then start apply the playground with the following command:
# To access services on remote clusters, add either --remote or --ingress-nginx --base-url=$yourdomain
kubectl run gitops-playground -i --tty --restart=Never \
  --overrides='{ "spec": { "serviceAccount": "gitops-playground-job-executer" } }' \
  --image ghcr.io/cloudogu/gitops-playground \
  -- --yes --argocd # additional parameters go here. 

# If everything succeeded, remove the objects
kubectl delete clusterrolebinding/gitops-playground-job-executer \
  sa/gitops-playground-job-executer pods/gitops-playground -n default  

In general docker run should work here as well. But GKE, for example, uses gcloud and python in their kubeconfig. Running inside the cluster avoids these kinds of issues.

Additional parameters

The following describes more parameters and use cases.

You can get a full list of all options like so:

docker run -t --rm ghcr.io/cloudogu/gitops-playground --help

Configuration file

You can also use a configuration file to specify the parameters (--config-file or --config-map). That file must be a YAML file.

Note that the config file is not yet a complete replacement for CLI parameters.

You can use --output-config-file to output the current config as set by defaults and CLI parameters.

In addition, For easier validation and auto-completion, we provide a schema file.

For example in Jetbrains IntelliJ IDEA, you can use the schema for autocompletion and validation when you put the following at the beginning of your config file:

# $schema: https://raw.githubusercontent.com/cloudogu/gitops-playground/main/docs/configuration.schema.json

If you work with an older version, you can use a specific git commit ID instead of main in the schema URL.

Then use the context assistant to enable coding assistance or fill in all available properties. See here for the full manual.

Apply via Docker

docker run --rm -t --pull=always -u $(id -u) \
    -v ~/.config/k3d/kubeconfig-gitops-playground.yaml:/home/.kube/config \
    -v $(pwd)/gitops-playground.yaml:/config/gitops-playground.yaml \
    --net=host \
    ghcr.io/cloudogu/gitops-playground --yes --argocd --config-file=/config/gitops-playground.yaml

Apply via kubectl

Create the serviceaccount and clusterrolebinding

$ cat config.yaml # for example
features: 
  monitoring:
    active: true

# Convention:
# Find the ConfigMap inside the current namespace for the config map
# From the config map, pick the key "config.yaml"
kubectl create configmap gitops-config --from-file=config.yaml

kubectl run gitops-playground -i --tty --restart=Never \
  --overrides='{ "spec": { "serviceAccount": "gitops-playground-job-executer" } }' \
  --image ghcr.io/cloudogu/gitops-playground \
  -- --yes --argocd --config-map=gitops-config

Afterwards, you might want to do a clean up. In addition, you might want to delete the config-map as well.

kubectl delete cm gitops-config 

Deploy Ingress Controller

In the default installation the GitOps-Playground comes without an Ingress-Controller.

We use Nginx as default Ingress-Controller. It can be enabled via the configfile or parameter --ingress-nginx.

In order to make use of the ingress controller, it is recommended to use it in conjunction with --base-url, which will create Ingress objects for all components of the GitOps playground.

The ingress controller is based on the helm chart ingress-nginx.

Additional parameters from this chart's values.yaml file can be added to the installation through the gitops-playground configuration file.

Example:

features:
  ingressNginx:
    active: true
    helm:
      values:
        controller:
          replicaCount: 4

In this Example we override the default controller.replicaCount (GOP's default is 2).

This config file is merged with precedence over the defaults set by

• the GOP and
• the charts itself.

Deploy Ingresses

It is possible to deploy Ingress objects for all components. You can either

• Set a common base url (--base-url=https://example.com) or
• individual URLS:

--argocd-url https://argocd.example.com 
--grafana-url