Modern Jenkins Unit 2 / Part 4: The Jenkins Plugin Image

The plugins image

Jenkins Plugins

You may have noticed that while we called the previous image jenkins-master, we never did drop the war in it. In fact, the only reference to that war we’ve seen is the very base image which sets a version, path, and a checksum. What’s the reason for this madness?

The answer is that the images we have built up until now are only a runtime environment for this image. The master image (the one we just built) will almost never change. When doing an upgrade the war never has new system requirements and rarely changes the directory structure or anything like that.

Jenkins Plugins Contents

What does change from deployment to deployment is the set of plugins, version of the Jenkins war, and the configuration that interacts with those things. For this reason I choose to run a vanilla Jenkins master container (with a few environment variable configs passed in) and a highly customized plugin container. This plugin container is where the binaries live and is volume mounted by the master to provide the software itself.

Let’s create it now and we can talk more about it after.


# images/jenkins-plugins/Dockerfile
FROM modernjenkins/jenkins-base
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location= \
      image_name=modernjenkins/jenkins-plugins \

# Add our plugin installation tool. Can be found here and is modified from the
# upstream version.
ADD files/ /usr/local/bin/

# Download the Jenkins war
RUN mkdir -p ${JENKINS_ROOT}/ref/warfile \
  && curl -fsSL ${JENKINS_URL} -o ${JENKINS_WAR} \
  && echo "${JENKINS_SHA}  ${JENKINS_WAR}" | sha256sum -c - \
  && chown -R ${user}:${user} ${JENKINS_ROOT}

# We will run all of this as the jenkins user as is dictated by the base imge
USER ${user}

# Install our base set of plugins and their depdendencies that are listed in
# plugins.txt
ADD files/plugins.txt /tmp/plugins-main.txt
RUN `cat /tmp/plugins-main.txt`

# Export our war and plugin set as volumes
VOLUME /usr/share/jenkins/ref/plugins
VOLUME /usr/share/jenkins/ref/warfile

# It's easy to get confused when just a volume is being used, so let's just keep
# the container alive for clarity. This entrypoint will keep the container
# running for... infinity!
ENTRYPOINT ["sleep", "infinity"]

You can see from the Dockerfile that this image is where the action is. We have a similar set of metadata at the top like the other images, then we add a file named This file is from the upstream Jenkins Docker image and it’s purpose is to install a set of plugins as well as any depdendencies they have. It can be downloaded from the link provided in the Dockerfile 1.

Then we go on to download the jenkins war and check it’s SHA. If the SHA does not match what we have in the base image, this step will fail and you know that something amiss is going on. Since the version and the SHA are both set in the very base image they should always match. There is never a scenario in which those two do not match up.

SHA 256 Sum

Once the war and tools are installed we can install our set of plugins. The script needs the war to run so now we should be ready. What this script is doing in the background is interacting with the Jenkins Update Center to attempt to install each plugin that is listed in plugins.txt. It will reach out to download the plugin and check for any depdendencies the plugin may have. If there are any, it will download those, resolve transitive deps and so on until the full set of plugins defined by us are installed along with any deps they need to function.

NOTE: This is different than the file that is out there. That script will not resolve dependencies and makes it very hard to audit which plugins you actually need.

PWD: ~/code/modern-jenkins/

# Add plugin resolver
cd images/jenkins-plugins
mkdir -p files/
wget -O files/ \
chmod +x files/

# Add a very base set of plugins to plugins.txt
# Add some credential storage
echo "credentials" >> files/plugins.txt
# Enable GitHub interactions
echo "github" >> files/plugins.txt
# Make our blue balls green
echo "greenballs" >> files/plugins.txt
# Give us Groovy capabilities
echo "groovy" >> files/plugins.txt

If you recall we discussed that this image is only going to provide the software itself and the Jenkins master image will provde the runtime. How that works is that we will export our plugins and warfile via the VOLUME statements at the bottom of this Dockerfile and mount them into the master via --volumes-from. This makes our plugins image an fully contained and versionable bundle of the master war and any plugins we need. A little later on, we will talk about how to include your configuration as well.

Finally we have the ENTRYPOINT. This version is farily simple: sleep infinity. What this does is keeps the container running even though we do not have a running process in it. Since this is only a container for our wars and JPIs, it doesn’t need to run the JVM or anything like that. It only needs to provide it’s exported volumes. If we were to omit the ENTRYPOINT, everything would still work as expected but for the fact that the jenkins-plugin container would not be running.

It would appear to be in a stopped state which for me is very confusing. The container is being used by the master (by way of volumes) and so it is indeed in use. The fact that Docker shows it as stopped is misleading IMO and so this just props up the container for clarity.

Building the image

Well, we’ve got another image to build and I think by this time you know what we’re going to do and it’s not DRY out our builders :P

PWD: ~/code/modern-jenkins/

# Warm up the copy machine...
cd images/jenkins-plugins
cp -rp ../jenkins-master/ .
perl -pi -e 's~jenkins-master~jenkins-plugins~g'

# Build the image
# yay! I worked on the first time :trollface:

Testing the image

As us rafters say, the proof is at the put-in. Let’s give it a whirl!

PWD: ~/code/modern-jenkins

# Start the plugins container first
docker container run --name plugins -d modernjenkins/jenkins-plugins

# Now fire up the master
docker container run --rm -ti --volumes-from=plugins -p 8080:8080 \
# Open the GUI
open http://localhost:8080

Jenkins Home

Would you look at that? Jenkins seems to be starting up swimmingly! If it is not for you, try to debug what exactly is going wrong. There are a lot of moving parts and this is a fairly complex system so don’t feel bad. It happens to all of us. Except us writing blog tutorials. We are always 100% right and our instructions work 11/10 times so you’re wrong and you should feel bad :P Seriously though, if something is jacked up in these instructions please use your super PR skills and help a brotha out by submitting a PR to the repo.

Unicorn Cleanup

Unicorn from:

Cleaning up

After running tests like these, we definitely need to begin thinking about cleanup. What would happen if we tried to run the same tests again right now? Feel free to try it, but the secret is that it won’t work. We need to delete the remnants from the previous test before starting another so I make it a habit to ensure a clean environmnet before I run a test and attempt to cleanup afterwards. The command I normally use for this is the equivalent of “nuke it ‘till it glows”: docker container rm -fv $(docker ps -qa). This little gem will remove all containers, running or not as well as any volumes they may have created (you may want to read more about that, volumes not in the state you thought they were can ruin your day in lots of ways).

One other thing you may be noticing is that no matter how diligent you are, you’re developing a stack of <none> images, weirdly named volumes, and orphaned networks. This is normal cruft left behind while doing Docker development and it can be removed by using docker system prune . This will remove:

  • all stopped containers
  • all volumes not used by at least one container
  • all networks not used by at least one container
  • all dangling images ('s)

NOTE: If you really want to clean up, add a -a and it will also remove images not attached to a running container. I find that to be annoying except when we’re in prod, but it is handy there.

(~) ------------------------------------------------------------------------- 🐳  unset (matt.bajor)
% docker rm -fv $(docker ps -qa)
(~) ------------------------------------------------------------------------- 🐳  unset (matt.bajor)
% docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y
Deleted Networks:

Deleted Images:
untagged: modernjenkins/jenkins-master@sha256:8f4b3bcad8f8aa3a26da394ce0075c631d311ece10cf7c23ce60058a9e47f6ed
deleted: sha256:96c78f549467f8b4697b73eddd9da299d8fd686696b45190a2bba24ad810529a
deleted: sha256:d1f38cb683287825bbf1856efdfaa87e2a7c279ceb793f9831b88b850ae1c9a0
deleted: sha256:5371c45cef2d3c5c468aae4fd5e93c335e8e681f2aa366f6122902c45e8ec9cb
deleted: sha256:079be452ec3e99b51a35b76e67b1bb3af649c3357e3ba05d2b6bd2a8127804b4
deleted: sha256:87baad26b39521ddd0d7b12ac46b2f92344f2f8ad34f0f35c524d5c0c566b409
deleted: sha256:c348763948964e1f63c427bea6b4d38c3a34403b61aee5e7b32059a3c095af32
deleted: sha256:6f92439bdac179e8c980dc6a7eb4f9647545e9c6d34d28edbba3c922efa9ea1e
deleted: sha256:edd5cbd4dc3cb3e9ab54bb1d7f446d5638c4543f04f2b63ae1a3e87a661be7a2
deleted: sha256:7890def677cf6649567c4355ef8f10c359f71c0ac9ca6ab94d8f359a5d57f84d
deleted: sha256:2704ec820811576ee2c60b8a660753939457f88fbe6938c2039489a6047ec59c
deleted: sha256:202acc3c794ce58a5e0b0e6b3285ab5ae27c641804c905a50b9ca7d5c601b2b3
deleted: sha256:70e19603643ce03f9cbff3a8837f1ebfb33fe13df7fba66c2501be96d9a2fb93
deleted: sha256:8e757cb858613c81e5fa8fb2426d22584539c163ce4ab66d6b77bd378ee2817a
deleted: sha256:18d1a064d790f3be371fef00813efe1c78996eab042977b952f4cbf067b846e8
deleted: sha256:bddcbf75436ff49e435fe3c371337b6b12ae125e68e0d833ac6180ffd82f34d9
deleted: sha256:f4dae60dcb2542e532eb05c94abba2da00d5a36360cb1d79cb32f87bf9b9c909
deleted: sha256:12f7c2589fdbb6e8b9ac78983511df70e9613c8da42edf23ee1cdb3599437233
deleted: sha256:26b155d41fabd6881f871945586c623a485688fc67f08223df288522f7aeed87
deleted: sha256:3a7c393698419b8f4f7a1464264459d2662d9015b3d577ad8cb12e0b4ae069a5
deleted: sha256:53794a3680b75ae98f70ab567db86a1b0e262615a8194bad534edfd5d8acc2f2
deleted: sha256:13449dedb3ec5df1f1b969aa2f1f93bb2a3bed2fb3ebe7279cce52b750696031
deleted: sha256:55aae84cda94b4611f73ec70b4cc1ea7ce4bbb77a1999b585fcc46c6239fe2a5
deleted: sha256:b41674288931c4e4bcd43e9fcc0d0af8d9ddd9a31f04506050ce0f0dfc59e3e3

Total reclaimed space: 313.9MB

Commit, push, PR

You know the drill. Integrate early, integrate often. Make sure you actually are looking at the work you’re merging. Afterall, it has your public name on it twice.

If you did get lost (I know I had to make a minor change to my base image), take a look at the unit2-part4 tag here:

Modern Jenkins Unit 2 / Part 3: Building the Jenkins Master Image

NOTE: Make sure you’re checking out a branch at the beginning of each section!

Building our master image

Jenkins Master

Now that we have a good base to inherit from, we can begin building out the rest of our images inheriting from that one. The next image we need is for the master. This image won’t contain too much other than generic configuration and a couple tools because we want our master image itself to be as generic as possible. The customization of each provisioned Jenkins master consists of configuration and plugins which we will package in a separate image. We will talk more about why it’s broken down this way later on. For now, let’s take a look at what we have for a Jenkins master image (modernjenkins/jenkins-master):


# images/jenkins-master/Dockerfile
FROM modernjenkins/jenkins-base
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location= \
      image_name=modernjenkins/jenkins-master \

# Jenkins' Environment

# `/usr/share/jenkins/ref/` contains all reference configuration we want 
# to set on a fresh new installation. Use it to bundle additional plugins 
# or config file with your custom jenkins Docker image.
RUN mkdir -p /usr/share/jenkins/ref/init.groovy.d

# # Disable the upgrade banner & admin pw (we will add one later)
RUN echo 2.0 > /usr/share/jenkins/ref/jenkins.install.UpgradeWizard.state \
    && echo 2.0 > ${JENKINS_HOME}/jenkins.install.InstallUtil.lastExecVersion

# Fix up permissions
RUN chown -R ${user} "$JENKINS_HOME" /usr/share/jenkins/ref

# Install our start script and make it executable
# This script can be downloaded from
COPY files/ /usr/local/bin/
RUN chown jenkins /usr/local/bin/* && chmod +x /usr/local/bin/*

# Make our jobs dir ready for a volume. This is where job histories
# are stored and we are going to use volumes to persist them
RUN mkdir -p ${JENKINS_HOME}/jobs && chown ${user}:${group} ${JENKINS_HOME}/jobs

# Install Docker (for docker-slaves plugin)
RUN yum-config-manager --add-repo \ \
    && yum makecache fast \
    && yum install -y docker-ce \
    && yum clean all -y

# Switch to the Jenkins user from now own
USER ${user}

# Configure Git
RUN git config --global "" \
    && git config --global "CI/CD LIfe Jenkins"

# Main web interface and JNLP slaves
EXPOSE 8080 50000
ENTRYPOINT ["/usr/local/bin/"]

Looking at this Dockerfile, you may see a few new things like USER (will run the commands after this declaration as the defined user) and EXPOSE (exposes defined ports for binding to an outside port), but for the most part it’s very similar to the previous one. Set a few ENV vars, RUN a few commands etc.

We need a build script so we’ll do the same thing that we did before (except now we have the script in our repo) by creating a that can also push. Let’s just duplicate this now:

PWD: ~/code/modern-jenkins/

cd images/jenkins-master
cp -rp ../jenkins-base/ .
perl -pi -e 's~jenkins-base~jenkins-master~g'

Now we have a nice little build script for this image too. While a puppy might have died when we copy/pasta’d I didn’t hear it whimper.

There is one more file that we need for this image and it’s the startup script. Since the internet was generous enough to provide one, we should just use it. This is the script that powers the official image and I’ve got a copy of it just for you in my repo. To retrieve it, use wget:

PWD: ~/code/modern-jenkins/

cd images/jenkins-master
mkdir files
wget -O files/ \
chmod +x files/

Build the image and test it out

Now that we’ve got all the files created that our image depends on, let’s build and test it a bit.

PWD: ~/code/modern-jenkins/

# Build it
cd images/jenkins-master

# Run it
docker container run --rm -ti modernjenkins/jenkins-master bash
docker version

# You should see the Docker client version only

Commit, push, PR

The master image seems to be gtg so let’s get it integrated. You may now be seeing what we mean by ‘continuous integration’. Every time we have a small chunk of usable work, we integrate it into the master branch. This keeps change sets small and makes it easier for everyone to incorporate the steady stream of changes into their work without spending days in Git hell.

You can compare your git tree to mine at state at the unit2-part3 tag here: The Docker images are also available to pull if you don’t feel like building them for some reason.

Our next move will be to build the meat of our system: the plugins container. Awwww Yeaahhhhh

Next Part: Building the Jenkins Plugin image

Modern Jenkins Unit 2 / Part 2: Building the Base Jenkins Image (and Intro to Docker)

Note: I am assuming familiarity with Docker for this tutorial, but even if you’ve never used it I think it should still be possible to follow along. It always helps to know your tools though so if you’re unfamiliar take some time to do a Docker Hello World or the like. It will be worth your investment in time as we will be using this technology throughout the tutorial. Everything we will do is based on Docker for Mac which you can download here: Linux users should be able to follow along without much adjustment too.

Building our Images

Under Construction

Well I think we’re fully setup with a great foundation at this point. We have a general spec for the system we would like to create, we have a nicely organized code repository hosted in GitHub and setup with a Grade A PR template that will ensure we’re thinking about what we’re doing, and we have a workflow that works for us and is reusable in any situation (nobody hates receiving PRs). It is time to actually begin writing some code!

Nearly every software vendor provides a Docker image for their piece of software which is super awesome when spiking things out or researching a new technology. The reality of it is though that a lot of companies have a security requirement that all software is vetted by the security team and then consumed from internal repositories. These repositories are served up by tools such as Artifactory 1 and feature built-in security scanning via Black Duck 2, permission models that allow only certain users to publish, and promotion mechanisms for getting only verifiable software into the environment. Pulling Docker images straight off of the Hub does not fit into that model at all.

Artifactory + Black Duck FTW

For that reason, we are going to develop a set of our own images with a common base. This gives us commonality between images which has many benefits including flexibility to add only the software that we want to. Now our examples will use public servers for all of this activity, but you can substitute those URLs for the URLs of your own internal artifact repository.

While we don’t want to trust every Docker image that has been published, we do have to start our chain of trust somewhere. In our case we will start with the CentOS 7 base image from the Docker Hub. There are lots of other great options out there, such as Alpine and Ubuntu, but I think CentOS is perfectly fine for this application and is what I use on a daily basis due to certain requirements.



Java (but not from you-know-who)

This image contains purely the JDK. Since we decided to base this image on CentOS (for security, support, compatibility, and reliability to name a few reasons) that is where our chain of trust begins. I personally have been trusting CentOS DVDs for an extremely long time so I feel confident they are a good place to start. On top of the Centos 7 base we will install the OpenJDK and setup a few environment vars. Let’s show the whole file and then talk about what each of the sections are.


# images/jenkins-base/Dockerfile
FROM centos:7
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location= \
      image_name=modernjenkins/jenkins-base \

# Jenkins' Environment
ENV JENKINS_HOME /var/jenkins_home
ENV JENKINS_ROOT /usr/share/jenkins
ENV JENKINS_WAR /usr/share/jenkins/ref/warfile/jenkins.war
ENV user=jenkins
ENV group=jenkins
ENV uid=1000
ENV gid=1000

# Jenkins Version info
ENV JENKINS_SHA d1ad00f4677a053388113020cf860e05a72cef6ee64f63b830479c6ac5520056

# These URLs can be swapped out for internal repos if needed. Secrets required may vary :)

# Jenkins is run with user `jenkins`, uid = 1000
# If you bind mount a volume from the host or a data container,
# ensure you use the same uid
RUN groupadd -g ${gid} ${group} \
    && useradd -d "$JENKINS_HOME" -u ${uid} -g ${group} -s /bin/bash ${user}

# Install our tools and make them executable
COPY files/jenkins-support /usr/local/bin/jenkins-support
RUN mkdir -p ${JENKINS_ROOT}
RUN chown jenkins /usr/local/bin/* ${JENKINS_ROOT} \
    && chmod +x /usr/local/bin/*

# Configure to Denver timezone. I dislike debugging failures in UTC
RUN unlink /etc/localtime && ln -s /usr/share/zoneinfo/America/Denver /etc/localtime

# Install Java, Git, and Unzip
RUN yum install -y java-1.8.0-openjdk-devel tzdata-java git unzip \
    && yum clean all

The above Dockerfile will be our base image that everything else will inherit from. While we are initially only creating a single Jenkins master, you may find that others in your organization would like their own Jenkins instance and this pattern ensures you’re ready for it without sacrificing readability. Now let’s talk about what is in this Dockerfile.


# images/jenkins-base/Dockerfile
FROM centos:7
MAINTAINER matt@notevenremotelydorky

LABEL dockerfile_location= \
      image_name=modernjenkins/jenkins-base \

This information is critical when tracking down a source in the supply chain as well as for new contributors who want to change how the container works.

  • # comment at the top is just the path within the repo to the file itself
  • FROM defines the image that we are building on top of
  • MAINTAINER tells who the maintainer of this image is
  • LABEL section provides labels that can be accessed with docker inspect


# Jenkins' Environment
ENV JENKINS_HOME /var/jenkins_home
ENV JENKINS_ROOT /usr/share/jenkins

These environment variables values that we want to have permanently baked into the image. They will be available in any container that is instantiated from this image or any other that inherits it. These types of variables make it easy to bring consistency across the environment.

Files & Commands (Actually doing the work)

RUN groupadd -g ${gid} ${group} \
    && useradd -d "$JENKINS_HOME" -u ${uid} -g ${group} -s /bin/bash ${user}

# Install our tools and make them executable
COPY files/jenkins-support /usr/local/bin/jenkins-support

These steps actually modify our image by installing software, modifying the filesystem, adding files from the build context, etc. They can use the ENV vars set above or arguments passed in as well as all other kinds of manipulations. You can see all the possible commands here:

Adding the jenkins-support file to the repo

We depend on a file called jenkins-support to make things work correctly. It is basically a shim to get Jenkins working within a Docker container properly. It cant be downloaded from my repo like so:

PWD: ~/code/modern-jenkins

cd images/jenkins-base
mkdir files
wget -O files/jenkins-supoort
chmod +x files/jenkins-support

Notes about images

Each line in a Dockerfile creates a layer and then all of these layers are mushed together (or was it squished?) to make our root fs. This mushing process is only ever additive so what that means is if you create a big file in one RUN step but then remove it in another RUN step, you’re not actually going to see any difference in image size. The key is finding the right balance between number of layers and size of layers. If we can keep layers under 50mb but still split up our logical process into easily understood and intuitive blocks (ie: doing a full yum transaction in one RUN block) then we’re sittin’ pretty.

From the Docker website Docker Image Layers

There is so much more I would like to tell you about best practices that I’ve found around image creation that I will have to save it for another post. Just know for now, we can never delete data that was created in a previous layer. That will directly translate into cleaning up after yourself in an atomic action. A real example is this:

# Install Java, Git, and Unzip then cleanup
RUN yum install -y java-1.8.0-openjdk-devel tzdata-java git unzip \
    && yum clean all

Building the image

Now that we have a super awesome Dockerfile, we need to build it. Normally I would have you do docker image build -t blah/boring . etc., but today I’m going to set your future self up for a win. We’re going to write a script right off the bat to build this thing. I promise you that you will be rebuilding this image at least 2 more times so let’s just go ahead and script it from the get go.


#!/bin/bash -el
# images/jenkins-base/

# Define our image name

# Accept any args passed and add them to the command
docker image build ${@} -t $image_name $(dirname -- "$0")

# If we add PUSH=true to the command, it will push to the hub
if [ "$PUSH" = true ] ; then
  docker image push $image_name

This will not be the last time we see this lil’ guy as we will add it to all of the image repos. Some may say “That’s not DRY Matt!”, to which I say “Suck a lemon!”. This code will never change and know that you can cd images/blah && ./ really makes it easy and convenient to work with these images. Now we run the script and out pops a baby Docker :)

PWD: ~/code/modern-jenkins

cd images/jenkins-base
chmod +x ./ # Gotta set executable perms
# ...
# profit!


yey! You’ve built your first Docker image (for this project)!

Testing the image

We can now go ahead and give this image a quick spin. It won’t be too exciting, but we can probably run the standard test to see that Java is installed:

PWD: ~/code/modern-jenkins

# Run the container and pop yourself into a shell
docker run --rm -ti modernjenkins/jenkins-base bash

# Check for java
java --version
# damn
java --help
# ugh
java version
# wtf! oh right...
java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-b16)
OpenJDK 64-Bit Server VM (build 25.141-b16, mixed mode)

Commit and push

OK, so now that we have a nice working image, this seems like a perfect place to call shippable increment. Let’s commit to our branch, push it to origin, clean up any erroneous commits by squashing. , and create a PR. We will then self review it, confirm everything looks up to snuff, and merge. Then a nice little git pull should get us all up to date locally and we can begin work on the next increment.

PWD: ~/code/modern-jenkins/

git checkout -b feat-add_jenkins-base_image
git add .
git commit -m "Add a base image containing OpenJDK 8"
git push origin feat-add_jenkins-base_image

Moving on

Now we’ll begin to build on top of our base images. If you need to see the repo’s state at the end of this section, please rever to the unit2-part2 tag here:

Next Part: Building the Jenkins Master image