My Bloody Jenkins - An opinionated Jenkins Docker Image

What's in the Box?

My Bloody Jenkins is a re-distribution of the Official LTS Jenkins Docker image bundled with most popular plugins and ability to configure most aspects of Jenkins from a simple and single source of truth represented as YAML.

The image can get the configuration from several data sources such as: File, S3, Environment Variable, HTTP, Kubernetes ConfigMap and Kubernetes Secret.

The image supports "Watching" configuration changes and applying them immediately without restarting jenkins.

The image is "Battle Proven" and serves as the base ground for several Jenkins deployments in production.

Features

• Configuration Coverage:
  • Security Realm (LDAP/AD/Simple Jenkins database)
  • Global Security Options
  • Authorization
  • Jenkins Clouds (Amazon ECS, Kubernetes, Docker)
  • Global Pipeline Libraries
  • Seed Jobs
  • JobDSL Scripts
  • Script approvals
  • Notifiers (Hipchat, Slack, Email, Email-Ext)
  • Credentials (aws, userpass, sshkeys, certs, kubernetes, gitlab, simple secrets)
  • Tools and installers (JDK, Ant, Maven, Gradle, SonarQube, Xvfb,Golang)
  • Misc. Plugins configuration such as Jira, SonarQube, Checkmarx, Artifactory
  • Misc. Configuration options such as Environment variables, Proxy
• Support additional plugins installation during startup without the need to build your own image
• Supports quiet startup period to enable docker restarts with a graceful time which Jenkins is in Quiet Mode
• Automated Re-Configure based on configuration data change without restarts
• Supports Dynamic Host IP configuration passed to clouds when Jenkins is running in a cluster
• Supports dynamic envrionment variables from consul and vault using envconsul
• Supports configuration-as-code-plugin as an alternative configuration syntax

Why Use the term "Bloody"?

The term "My Bloody Jenkins" came from the fact that I tried to put all my "battle" experience, (i.e. blood, sweat and tears) within the image. I just thought it is a "catchy" name for this kind of a repository.

Demo

A step by step demo can be found here

k8s Helm Chart

In order to deploy in k8s, A corresponding Helm Chart can be found Here

helm repo add odavid https://odavid.github.io/k8s-helm-charts
helm install odavid/my-bloody-jenkins [-f values.yml]

Some Usage Examples

• docker-plugin cloud cloud using Docker Plugin cloud with seed job. See examples/docker
• kubernetes cloud using Minikube with seed job. See examples/kubernetes

Releases

Docker Images are pushed to Docker Hub

Each release is a git tag v$LTS_VERSION-$INCREMENT where:

• LTS_VERSION is the Jenkins LTS version
• INCREMENT is a number representing that representing the release contents (i.e additional configuration options, bugs in configuration, plugins, etc...)

For each git tag, there following tags will be created:

• $LTS_VERSION-$INCREMENT - one to one releationship with git tag
• $LTS_VERSION - latest release for that LTS version
• lts - represents the latest release

Each master commit, will be tagged as latest

# get the latest release, alpine
docker pull odavid/my-bloody-jenkins:lts
# get the latest debian release
docker pull odavid/my-bloody-jenkins:lts-debian
# get the latest jdk11 release
docker pull odavid/my-bloody-jenkins:lts-jdk11

# get the latest 2.164.1 LTS
docker pull odavid/my-bloody-jenkins:2.164.1
# get the latest 2.164.1 debian LTS
docker pull odavid/my-bloody-jenkins:2.164.1-debian
# get the latest 2.164.1 jdk11 LTS
docker pull odavid/my-bloody-jenkins:2.164.1-jdk

# get a concrete 2.164.1 release
docker pull odavid/my-bloody-jenkins:v2.164.1-109

# get the latest unstable image (alpine)
docker pull odavid/my-bloody-jenkins
# get the latest unstable debian image
docker pull odavid/my-bloody-jenkins:debian
# get the latest unstable jdk1 image
docker pull odavid/my-bloody-jenkins:jdk11

Alternate docker registry

The docker image is also published to ghcr.io So you can also pull it from ghcr.io/odavid/my-bloody-jenkins:<tag>

Environment Variables

The following Environment variables are supported

• JENKINS_ENV_ADMIN_USER - (mandatory) Represents the name of the admin user. If LDAP is your choice of authentication, then this should be a valid LDAP user id. If Using Jenkins Database, then you also need to pass the password of this user within the configuration.

• JAVA_OPTS_* - All JAVA_OPTS_ variables will be appended to the JAVA_OPTS during startup. Use them to control options (system properties) or memory/gc options. I am using few of them by default to tweak some known issues:

  • JAVA_OPTS_DISABLE_WIZARD - disables the Jenkins 2 startup wizard
  • JAVA_OPTS_CSP - Default content security policy for HTML Publisher/Gatling plugins - See Configuring Content Security Policy
  • JAVA_OPTS_LOAD_STATS_CLOCK - This one is sweet (: - Reducing the load stats clock enables ephemeral slaves to start immediately without waiting for suspended slaves to be reaped

• JENKINS_ENV_CONFIG_YAML - The configuration as yaml. When this variable is set, the contents of this variable can be fetched from Consul and also be watched so jenkins can update its configuration everytime this variable is being changed. Since the contents of this variable contains secrets, it is wise to store and pass it from Consul/S3 bucket. In any case, before Jenkins starts, this variable is being unset, so it won't appear in Jenkins 'System Information' page (As I said, blood...)

• JENKINS_ENV_CONFIG_YML_URL - A comma separated URLs that will be used to fetch the configuration and updated jenkins everytime the change. This is an alternative to JENKINS_ENV_CONFIG_YAML setup. Supported URLs:

  • s3://<s3path> - s3 path
  • file://<filepath> - a file path (should be mapped as volume) - can be a file, folder or glob expression (e.g. file:///dir/filename or file:///dir or file:///dir/*.yml)
  • http[s]://<path> - an http endpoint

Note: If multiple URLs are passed or the file url contains a dir name or a glob expression, all yaml files are being deep merged top to bottom. This behavior enables to separate the configuration into different files or override default configuration.

• JENKINS_ENV_CONFIG_YML_URL_DISABLE_WATCH - If equals to 'true', then the configuration file will be fetched only at startup, but won't be watched. Default 'false'

• JENKINS_ENV_CONFIG_YML_URL_POLLING - polling interval in seconds to check if file changed in s3. Default (30)

• JENKINS_ENV_HOST_IP - When Jenkins is running behind an ELB or a reverse proxy, JNLP slaves must know about the real IP of Jenkins, so they can access the 50000 port. Usually they are using the Jenkins URL to try to get to it, so it is very important to let them know what is the original Jenkins IP Address. If the master has a static IP address, then this variable should be set with the static IP address of the host.

• JENKINS_ENV_HOST_IP_CMD - Same as JENKINS_ENV_HOST_IP, but this time a shell command expression to fetch the IP Address. In AWS, it is useful to use the EC2 Magic IP: JENKINS_ENV_HOST_IP_CMD='curl http://169.254.169.254/latest/meta-data/local-ipv4'

• JENKINS_HTTP_PORT_FOR_SLAVES - (Default: 8080) Used together with JENKINS_ENV_HOST_IP to construct the real jenkinsUrl for jnlp slaves.

• JENKINS_ENV_JENKINS_URL - Define the Jenkins root URL in configuration. This can be useful when you cannot run the Jenkins master docker container with host network and you need it to be available to slaves

• JENKINS_ENV_ADMIN_ADDRESS - Define the Jenkins admin email address

• JENKINS_ENV_PLUGINS - Ability to define comma separated list of additional plugins to install before starting up. See plugin-version-format.

This is option is not recommended, but sometimes it is useful to run the container without creating an inherited image.

• JENKINS_ENV_QUIET_STARTUP_PERIOD - Time in seconds. If speficied, jenkins will start in quiet mode and disable all running jobs. Useful for major upgrade.

• JENKINS_ENV_CONFIG_MODE - If set to jcasc, then Configuration as Code Plugin will be used instead of Built-in Configuration Handlers. See JCasC Demo.

This option will disable all configuration handlers used by the image! If you still want to use builtin configuration handlers, together with dynamic JCasC snippets, please see Configuration as Code Section.

Configuration Reference

The configuration is divided into main configuration sections. Each section is responsible for a specific aspect of jenkins configuration.

Environment Variables Section

Responsible for adding global environment variables to jenkins config. Keys are environment variable names and values are their corresponding values. Note that variables names should be a valid environment variable name.

  environment:
    ENV_KEY_NAME1: ENV_VALUE1
    ENV_KEY_NAME2: ENV_VALUE1

Environment variable Substitution and Remove Master Env Vars

You can use ${ENV_VAR_NAME} within the config.yml in order to use environment variables substitution for sensitive data (e.g k8s secrets). When you pass secrets environment variables to the container, Jenkins will display them in the 'System Info' page. In order to disable that beheviour, you can use remove_master_envvars section and add regular expressions for variables you don't want to show on the SystemInfo page.

Escaping ${VAR} to be used as is without substitution, is done by using \${VAR} within the yaml file

security:
  realm: ldap
  managerDN: cn=search-user,ou=users,dc=mydomain,dc=com
  managerPassword: '${LDAP_PASSWORD}' # Use LDAP_PASSWORD environment variable
  ...

remove_master_envvars:
  - '.*PASS.*'
  - '.*SECRET.*'
  - 'MY_SPECIAL_VARIABLE'

Environment Variables Data Sources

The image supports the following data sources for environment variables:

• Native - Environment variables that are passed to the container at startup
• Files - By passing ENVVARS_DIRS variable to the container, selected directories can be treated as environment variable source
• Consul - using envconsul
• Vault - using envconsul

Environment Variables Values From Files

When using Environment Variable Substitution within the config.yml file, you can consume environment variables values directly from files contents within folders. This is useful especially when using k8s secrets volume mappings

In order to activate this feature, you need to pass ENVVARS_DIRS variable to the container with a comma separated list of directories.

Example

Assuming you have the following files within the container:

• /var/secret/username
• /var/secret/password
• /var/other-secret/ssh-key
• /var/other-secret/api-token

Setting the following ENVVARS_DIRS environment variable as follows:

ENVVARS_DIRS=/var/secret/,/var/other-secret

Will produce the following environment variables:

• SECRET_USERNAME - contents of /var/secret/username
• SECRET_PASSWORD - contents of /var/secret/password
• OTHER_SECRET - contents of /var/other-secret/ssh-key
• OTHER_SECRET_API_TOKEN - contents of /var/other-secret/api-token

Note that variable names are the <FOLDER_NAME>_<FILE_NAME> sanitized and uppercased

Using envconsul to Fetch Dynamic Environment Variables from Consul and Vault

When using Environment Variable Substitution within the config.yml file, you can direct the container to automatically fetch them from from consul and vault using envconsul

The following environment variables need to be provided in order to support it:

• ENVCONSUL_CONSUL_PREFIX - Comma separated values of consul key prefixes - Mandatory if using consul to fetch information
• CONSUL_ADDR - Consul address (host:port) - Mandatory if using consul to fetch information
• CONSUL_TOKEN - Consul ACL Token - The token that used to be authorize the container to fetch the keys from consul - Mandatory if consul ACLs are in use
• ENVCONSUL_VAULT_PREFIX - Comma separated values of vault key prefixes - Mandatory if using vault to fetch information
• VAULT_ADDR - Vault address (http[s]://host:port) - Mandatory if using vault to fetch information
• VAULT_TOKEN - Vault ACL Token - The token that used to be authorize the container to fetch the keys from vault - Mandatory
• ENVCONSUL_UNWRAP_TOKEN - true/false (default = false), see - tells Envconsul that the provided token is actually a wrapped token that should be unwrapped using Vault's cubbyhole response wrapping
• ENVCONSUL_MAX_RETRIES - (default = 5), How many time the envconsul will retry to fetch data
• ENVCONSUL_ADDITIONAL_ARGS - A list of command line arguments to append to the envconsul CLI. For more details, please read the envconsul READM

The following parameters are being added to the envconsul CLI:

• -sanitize - replaces all invalid characters to underscore
• -upcase - All keys will become Uppercase

Due to An open Issue with envconsul and vault > 0.9.6, Only Vault versions <= 0.9.6 can be used

Security Section

Responsible for:

• Setting up security realm
  • jenkins_database - the adminPassword must be provided
  • ldap - LDAP Configuration must be provided
  • active_directory - Uses active-directory plugin
  • saml - Uses saml plugin
  • google - Uses google-login plugin
  • oic - Uses oic-auth plugin
  • github - Uses github-oauth plugin
• User/Group Permissions dict - Each key represent a user or a group and its value is a list of Jenkins Permissions IDs
  • For disable configure Matrix based Security you should add "unsecureStrategy: true" (Anyone can do anything)

# jenkins_database - adminPassword must be provided
security:
    realm: jenkins_database
    adminPassword: S3cr3t
    # When using jenkins_database, you can also create