Kubernetes configs

Intro

Advanced Kubernetes YAML configurations & templates, based on my experiences running Kubernetes in production at different companies.

The top-level directory contains standard Kubernetes object templates with many Best Practices, Tips & Tricks learned over time across production environments.

The sub-directories contain ready-to-run real world apps that I've run across environments.

Templates

Start with deployment.yaml / statefulset.yaml, for advanced users see kustomization.yaml.

The service.yaml and ingress.yaml configs contain settings for using static public IP addresses and locking down your cloud load balancer's firewall rules eg. to private IP addresses, and patches for Cloudflare Proxied or VPN IPs. You may need to extend those IP lists to your office / VPN / public addresses if really want to permit direct internet access to your ingresses and aren't proxying them through a WAF in proxied mode etc.

See also the Dockerfile template in the HariSekhon/Templates repo to containerize your custom apps for deploying on to Kubernetes.

Apps

Real-world app deployments are found in the more specific <app>/ directories.

These follow the standard Kustomize <app>/base/ and <app>/overlay/ layout to make it easy to use as-is by just tweaking a couple settings in the overlay to your specific environment.

CI/CD

Advanced auto-scaling production-grade CI/CD on Kubernetes:

ArgoCD - deployment, configs and optimizations. Start here: argocd/base/kustomization.yaml
Jenkins - jenkins server and dynamically scaling agents on kubernetes. Start here: jenkins/base/kustomization.yaml
- see also: Jenkins repo with advanced Jenkinsfile & Jenkins Shared Library
TeamCity - teamcity server and dynamically scaling agents on kubernetes. Start here: teamcity/base/kustomization.yaml
Selenium Grid - simple and distributed auto-scaling deployments. Start here: selenium-grid/base/kustomization.yaml / selenium-grid-distributed/base/kustomization.yaml

Helm Repos

Helm repos you should probably have installed:

Repo Label	URL
stable	https://charts.helm.sh/stable
bitnami	https://charts.bitnami.com/bitnami
fairwinds-stable	https://charts.fairwinds.com/stable

Table generated from helm-repos.txt by script generate_repos_markdown_table.sh

These repos can quickly installed in one command using script install_repos.sh:

./install_repos.sh

Helm + Kustomize integration

See kustomization.yaml for 2 methods provided:

template the Helm chart using a values.yaml to Git and serve from there (see DevOps Bash Tools for the helm_template.sh convenience script)
dynamically load the Helm chart from upstream with a values.yaml

...then patch override anything the chart doesn't directly support using the standard Kustomize patching examples given in the kustomization.yaml.

Production Ready Checklist

Healthchecks - readiness/liveness probes, see deployment.yaml
Horizontal Pod Autoscaler - horizontal-pod-autoscaler.yaml
Pod Disruption Budget - pod-disruption-budget.yaml
Pod Anti-Affinity - stable vs preemptible, HA across AZs, see deployment.yaml
Ingress Controllers - Nginx (config), Kong (config) or Traefik (config)
Ingress SSL - Cert Manager (config) for Automatic Certificate Management using the popular free Let's Encrypt certificate authority
App Lifecycle Management - ArgoCD (config)
App Ingresses - ingress.yaml, */base/ingress.yaml
App Resources - see resources section in deployment.yaml
App Right-Sizing - Goldilocks (config) to generate VPAs and resource recommendations
DNS - External DNS (config) integration to AWS Route53, Cloudflare etc.
Secrets - External Secrets (config) integration to AWS Secrets Manager, GCP Secret Manager etc. or Sealed Secrets (config)
Resource Quotas per Namespace - resource-quota.yaml
Limit Ranges - per object limits within a namespace - limit-range.yaml
Network Policies - network-policy.yaml
Pod Security Policies - pod-security-policy.yaml
Governance, Security & Best Practices - Polaris (config) for recommendations
Find Deprecated API objects to replace - Pluto - see pluto_detect_kustomize_materialize.sh, pluto_detect_helm_materialize.sh and pluto_detect_kubectl_dump_objects.sh in the DevOps Bash Tools repo
Helm is not IaC idempotent by itself - that is PoC territory - you must wrap it in Kustomize, ArgoCD or similar to detect live drift!
Quickly update any Helm Charts in a kustomization.yaml file using kustomize_update_helm_chart_versions.sh in the DevOps Bash Tools repo

Further Documention

The best documentation links are provided at the top of each yaml for fast referencing (my advanced .vimrc can open these URLs from the current file via a hotkey!)

See also HariSekhon/Knowledge-Base, especially:

Extra Docs

Datree Kubernetes ArgoCD best practices

Environment Enhancements

.envrc - use with direnv to auto-load correct Kubernetes context isolated to current shell to avoid race conditions between shells and scripts caused by naively changing the global ~/.kube/config context

Shortcut symlinks are for faster instantiation from these configs using the standard kubernetes shortcuts such as new pvc.yaml - see the Templates repo for more details on the new command to fast create new files from templates.