Modern Jenkins Unit 2 / Part 1: GitHub + GitHub Flow

Storage + Process

Now that we have a good idea of what the desired traits and abilities of our system should be, we can begin to lay down the foundation. The very beginnings of all projects start with a SCM (source control management) repository.

GitHub Logo

Multi or single repo?

We want our repo to be intuitively laid out as there is the chance that almost every engineer in the company will need to interact with it at some point. This means that we should separate concerns within the tree and make things easy to find. In the past we tended to use a lot of separate repos and to treat them as if they were fully independent and could be shared with any other team as is. I am a fan of this idea for certain use cases, but what I have found is that it makes things very hard to find on a system that you contribute to infrequently. I prefer instead to start out with a mono-repo and if there comes a time in which we need to break it up, do it then. Making a bunch of different repos for a project like this right off the bat is a premature optimization IMO.


I’m going to treat this as a tutorial so you should be able to follow along by running what’s in the code blocks

Flesh out the repo structure

First let’s make sure that we have a repo to work on setup in GitHub. You should be able to do this for free as GitHub allows unlimited open source projects. Once we get to the point of configuring our deployment, we will add some encryption to cover the secret data.

Create the repo and populate the master branch

GitHub Logo

Create an account on GitHub 1 if you do not have one already. GitHub is the worlds largest hub for sharing code and lots of companies use it as their primary SCM host with either their public or on-premise offering. We will be storing everything we do in this tutorial on GitHub. You can sign up for free (no credit card needed) here:

and the creating your first public repo:

NOTE: Don’t check the “Initialize this repo with a README” button.

You can name it anything you want, but I have named mine modern-jenkins. Clone this somewhere locally (I use ~/code for all of my repos) and let’s begin initializing the repo. I prefer to start with an empty commit when creating a new repo as we can then PR everything that ever hits the master branch. You can do this like so:

# Setup your Git author information first:
git config --global
git config --global "Matt Bajor"

# Navigate to your code directory
cd ~/code

# I have created a Github project here:
git clone
cd modern-jenkins

# Populate the master branch. This will be our first and ONLY commit directly to
# master. Everything else will go through a Pull Request
git commit --allow-empty -m "Initial Commit"
git push origin master

Begin with our GitHub Flow workflow

We will be using GitHub Flow 2 to work with our repo. Visit the link if you want to get a good idea of how it works, but the gist of it is that the master branch is where stable code lives. It should always be releasable. When we want to make a change, we will create a feature branch, make changes, then Pull Request 3 the changes back into master. Since the branches only live until the change is merged into master, they are considered to be “short lived”. There are some other workflows such as Git Flow 4 that work differently with “long lived” branches and other workflows, but they can lead to more of a headache with large, long running projects in my experience.

Our workflow will be:

  1. Cut a feature branch off of master
  2. Make changes to the branch to implement the feature
  3. Push your changes to the remote (GitHub)
  4. Create a Pull Request to merge your changes into master
  5. (self-)Review the changes and comment and/or approve
  6. Merge changes into master and delete the feature branch
  7. Pull the new master locally

Create repo skeleton

Let’s begin creating our repo!

PWD: ~/code/modern-jenkins/

# Starting from our empty master, checkout a feature branch
git checkout -b feat-repo_skeleton

# Create a base readme
echo '# Dockerized Jenkins Build System' >

# Create a place for our images
mkdir -p images/ && echo '# Docker Images' > images/

# Create a place for our deployment config
mkdir -p deploy/ && echo '# Deployment Scripts' > deploy/

# A spot for the DSL
mkdir -p dsl/ && echo '# Jenkins Job DSL' > dsl/


Whenever we change the master branch we will be submitting a Pull Request against the repo. GitHub gives you the ability to template the contents of the Pull Request form. Since we’re probably working by ourselves here, you may wonder “WTH Matt, why are we setting up collaboration tools when I’m clearly doing this tutorial by myself?” There are a couple of reasons:

  1. Get used to the process: As the system evolves and as it grows there will be more and more people adding to this repo. If we start off pushing to master it’s much easier to continue that tradition and end up with everyone pushing to master all willy nilly. If we start off with a robust process, it has a habit of sticking around.

  2. Fighting complacency: Since we’ll be self-reviewing most of this code, it can be really easy to just click ‘Merge’ without thinking of what you’re doing. This PR template has a small checklist that will keep you honest with yourself. If you know you have a tendency to skip over something go go ahead and add that to the checklist too.

  3. Change management: Going from working app to working app requires keeping an eye on what’s changing whenever they change. When things do go awry (and they will), PRs will help untangle the mess much quicker than a steady stream of commits to the master branch. In theory, it is much easier to tell when a PR breaks a repo instead of a single commit in the history.

  4. Thinking about your work in chunks: We really will be adding just a set of commits to a repo to get to our final form, but if we treat our work like that it’s never done. Instead, we should think about small chunks of work that bring value and can be deployed as a whole. Agile calls this a releasable increment. These chunks should make it easier to reason about what impact the change may have.

Even if I haven’t convinced you that this is important, I’m going to put it in a code block which will force your hand anyways. Ha!

cat <<EOF >
#### What does this PR do?

#### Why did you take this approach?

#### Contribution checklist

- [ ] The branch is named something meaningful
- [ ] The branch is rebased off of current master
- [ ] There is a single commit (or very few smaller ones) with a [Good commit message]( that includes the issue if there was one
- [ ] You have tested this locally yourself

#### A picture of a cute animal (optional)
<img src="" width="350"/>

Integrating our changes into master

Now that we have a layout we like and a fancy new PR template, let’s get them into the master branch. We will do this via PR.

PWD: ~/code/modern-jenkins/

# Add the changes to our branch
git add .
git commit -m "Create initial repo structure and boilerplate"
git push origin feat-repo_skeleton

Now that the changes are pushed up, browse to your repo in your favorite web browser. Mine is located at You should see a green button that says “Compare & Pull Request”:

Compare & pull request

Click that and it will take you to a form that allows you to set the title of the PR as well as a description. Enter both and click “Create Pull Request”. Feel free to describe exactly what you’re doing and why. It’ good practice ;)

Once it has been submitted, you can then view the changeset, comment on it if you like (don’t feel bad talking to yourself), and approve or modify the PR. Since this guy is pretty simple, take a quick look through your commit and make sure there are no typos etc. If all looks good, give er’ a ‘LGTM’ in the comments section and merge away.

NOTE: I always recommend disabling merge commits in a repo’s settings. These just muddy up the commit history and instead I prefer to use the “Squash Merging” setting instead. Disable merge commits This will squash all commits in the PR down to one and will allow you to edit the commit message before doing so. This really makes rebases and other git surgery easier than when there are a ton of merge commits to wade through. You can also do this before creating the PR if you like. See my post here on cleaning up your git history.

Sweet! You have made your first change on a road of many and you have done it in a very sustainable way that gives everyone working on the project context of what you’re doing and why. Most importantly, you’ve given your future self some really good reminders as to what you were thinking at the time the change was made.

On to the next chapter!

Now that we have our skeleton there, let’s begin hashing out the actual stuff in the repo :) If you need it, the code from this part can be found under the unit2-part1 tag here:

Next Post: Building Jenkins’ base Docker image (and brief Intro to Docker)

Modern Jenkins Unit 1 / Part 2: Architecting a Jenkins + Docker CI System

Defining the architecture requirements

We have defined the characteristics of our system so we know how we want it to behave. In order to obtain that behavior though, we need to determine what architectural features will get us there. Let’s go through our business requirements one by one and come up with some technical requirements to guide us through the project.

Easy to develop

We need a way to make changes to the system in order to move it forward that is easy to work with and as simple as possible. Developing in production may seem easy at first, but over time this creates a lack of trust in the system as well as a lot of downtime. Instead, let’s implement these technical features to make changing this system sustainable:

  1. New developers can get going by cloning the repo and running a command
  2. Everything can be run locally for development purposes
  3. Resetting to a known good state is quick and easy

Getting started workflow

PWD: ~/code

# Ok, let's get you setup FNG
git clone

cd modern-jenkins/deploy/master/

# You should be all set! You can begin development now.

Safe to develop

The nature of a CI system is to verify code exhibits the qualities we want and expect it to then deliver it production. This makes it inherently dangerous as it is extremely powerful and programmed to deliver software to production. We need the ability to modify this system without interfering with production by accident. To accomplish this we will implement:

  1. Splitting development and production credentials
  2. Disabling potentially dangerous actions in development
  3. Explicitly working with forks when possible

Dev vs. Prod

Consistently deployable

This system, when deployed to production, will be used by a large portion of your development team. What this means is that downtime in the build system == highly paid engineers playing chair hockey. We want our system to always be deployable in case there is an issue with the running instance and also to be able to quickly roll back if there is an issue with a new version or the like. To make this happen we will need:


  1. A versioning system that keeps track of the last known good deployment
  2. State persistence across deploys
  3. A roll-forward process
  4. A roll-back process

Easy to upgrade

The nature of software development involves large amounts of rapid change. Within our system this mostly consists of plugin updates, Jenkins war updates, and system level package updates. The smaller the change during an upgrade, the easier it is to pinpoint and fix any problems that occur and the way to do that is frequent upgrades. To make this a no-brainer we will:

  1. Build a new image on every change, pulling in new updates each time
  2. Automatically resolve plugin dependencies
  3. Smoke test upgrades before deployment

Dev vs. Prod

Programmatically configurable

Hand configuring complex systems never leads to the same system twice. There are lots of intricacies of the config including: order of operations, getting the proper values, dealing with secrets, and wiring up many independent systems. Keeping track of all this in a spreadsheet that we then manually enter into the running instance will get it done, but becomes pure nightmare mode after the first person is done working on it. Luckily with Jenkins’ built in Groovy init system 1, we will be able to configure Jenkins with code in a much more deterministic way. Most of this functionality is built in, but we will still need:

I heard you like CI/CD

  1. A process to develop, test, and deploy our Groovy configuration
  2. A mechanism to deploy it along with the master
  3. The ability to share these scripts across environments


With a consistently deployable, programmatically configured system it is almost as easy to redeploy instead of modifying the running instance. It is indeed easier to just click a checkbox in the GUI to change a setting the first time you do it, but the second, third, and fourth times become increasingly more difficult to remember to go check that box. For this reason, we are NEVER* going to mutate our running instance. We will instead just deploy a new version of our instance. This will only be a process change as the architectural features listed above will enable this process. We should however, document this process so that it will be followed by all.

  1. Documentation of how to get changes into production

100% in SCM

Everything we’re doing is ‘as code’ so we will store it in the source repository ‘as code’. This will include master configuration, secrets, job configuration, infrastructure provisioning and configuration, as well as anything else the system needs to operate. To support this we’ll need:

  1. A SCM repo (we’re using a Git repo hosted by GitHub)
  2. A layout that supports: - Docker containers - Deployment tooling - Job DSL - Secrets
  3. A mechanism for encrypting secrets in the repo

Git Repo Diagram

├── ansible
│   ├── playbook.yml
│   └── roles
├── deploy
│   ├── master
│   │   └── docker-compose.yml
│   └── slaves
├── images
│   ├── image1
│   │   └── Dockerfile
│   └── image2
│       └── Dockerfile
└── secure
    ├── secret1
        └── secret2


This pipeline can be considered the production line at a factory. It is where the individual components of your application are built, tested, assembled, packaged, and deployed to your customers. Any system that does the above mentioned processes should be heavily locked down as it is integral to the functioning of the company. Specifically we should:

Datacenter Cages

  1. Keep secrets secure and unexposed
  2. Segregate actions by user (both machine and local)
  3. Create an audit trail for any changes to the production system
  4. Implement least-privilege access where necessary
  5. Apply patches as soon as is feasible


If scalability is thought of from day 0 on a project it is much easier to implement than if it is bolted on later. These systems have a tend to grow over time and this growth should be able to happen freely. This is not always an easy task, but we will attempt to make it easier by:

  1. Keeping individual components de-coupled for independent scaling
  2. Not locking ourselves into a single platform or provider
  3. Monitoring the system to point out bottlenecks

Scaling to infinity… and beyond! Scaling to infinity

Now let’s start making some stuff!

Next Post: Code Repository and Development Lifecycle for a Jenkins + Docker CI/CD System

Modern Jenkins Unit 1 / Part 1: Planning a Jenkins + Docker CI System

What are we doing here?

I have been introduced to far more installations of Jenkins that suck than don’t. One of the issues with a system of this level of complexity that is shared between many teams is that they tend to grow organically over time. Without much “oversight” by a person or team with a holistic view of the system and a plan for growth, these build systems can turn into spaghetti infrastructure the same way a legacy codebase can turn into spaghetti code.

Devil Jenkins!

Eventually failures end up evenly dividing themselves amongst infrastructure, bad commits, and flaky tests. Once this happens and trust is eroded, the only plausible fix for most consumers is to re-run the build. This lack of trust really makes it less fun to develop quality, reliable software. Its very important to have trust that the systems you are using to verify your work and not the other way around. An error in a build should unambiguously point to the code, not the system itself.

While not all companies can afford to have a designated CI team supporting the build infrastructure, it is possible to design the system initially (or refactor) in a way that encourages good practice and stability thus elevating developers’ confidence in the system and speed with which they can deliver new features. This series of posts will hopefully be able to get you started in the right direction when having to build or refactor a CI / CD system using Jenkins.

Describing the characteristics of the end state

I am a big fan of Agile software development1 myself. I don’t believe that there is one kind of ‘Agile’ that works for everyone or anything like that, but I do believe 100% in the iterative approach agile takes to solve complex problems. Breaking work into small deliverables, and using those chunks for planning at multiple intervals such as:

  • Yearly: Very high level direction
  • Quarterly: General feature requirements
  • Sprint: Feature implementation details

Agile Lifecycle

When you have multiple perspectives on the scope and movement of a project, it really gives you the ability to manage expectations and make the most of your time. Since you have a general idea of what the long term goals are, you (ideally) can then make trade-off decisions with accurate values on the scales. This leads to less rewrites and the ability to put a bit more into the code you write because you know it will become legacy and you know Future You will appreciate the effort.

This is perhaps in opposition to the idea of hacking your way to a solution which is also valuable process, but more for a problem with an undefined domain. Luckily shipping software has most of problem domain defined so we’re able to set long, medium, and short time goals for our CI / CD infrastructure.

Anyways, let’s state some of the properties we want from our build system:

Super happy developer guy

  • Easy to develop: A common problem with complex systems is that the process of setting up a local development environment is just as complex, if not more so. We will be developing just about the entire system locally on our laptop so we should hopefully get this for minimal cost.

  • Safe to develop: Another property that goes hand in hand with easy to develop is safe to develop. If our local environment is always reaching out to prod to comment on PRs or perhaps pushing branches and sending Slack notifications, it can be misleading and confusing to figure out where exactly these things are coming from. We will attempt to null out any non-local side effects in our development environment to provide a safe place to work.

  • Consistently deployable: Hitting the button and getting an error during a deployment is one of the worst feelings ever. In most cases when we need to deploy, we need to deploy right now! While building this project we will always keep master releasable and work with our automation in a way that no matter when we need to deploy, it will go.

    Roll safe my friend

  • Easy to upgrade: The best way to mitigate dangerous upgrades is to do them more frequently. While this may sound counterintuitive, it works for a couple of reasons:
    1. The delta is smaller the more frequently you upgrade (and deploy!)
    2. Each time you upgrade and deploy, you get more practice at it
    3. If you fix a small error each upgrade, eventually most errors are fixed :) As we will see, what enables easy upgrades is a rollback safety net and a good way to verify the changes are safe. Since we’ll never be able to catch all the bugs, having the rollback as a backup tool is clutch.

    It's Groovy baby, groovy yeah!

  • Programmatically configurable: This is a giant one. Humans are really terrible at doing or creating the same thing over and over. This is way worse once you have a group of humans trying to do the same thing over and over (like creating jobs or configuring the master). Since we cannot be trusted to click the right buttons in the right sequence, we need to make sure the system does not depend on us doing that. We will cover a variety of tools to configure our system including the Netflix Job DSL, Groovy configuration via the Jenkins Init system, and later on Jenkinsfiles. There should be nothing that is hand jammed!

    Revision Control like a boss

  • Immutable: “Immutable Infrastructure” 2 is a term that was coined by Chad Fowler back in 2013. He proposes that the benefits of immutability in software architecture (like what is brought by functional programming) would also apply to infrastructure. What this translates to is that instead of mutating a running server by applying updates, rebooting, and changing configuration on running machines, when changes need to be made we should just redeploy a new set of servers with the updated code instead. This makes it much easier to reason about what state a system is in. You will see that this is one of the main characteristics that makes this Jenkins infrastructure different (and I think much better) than a legacy installation.

  • 100% in SCM: Since we are committing to programmatic configuration, it is crucial that we keep everything within source control management. We should apply the same rigor with our CI / CD system’s codebase that we do with the one it is testing. PRs and automated tests should be the norm with anything you are creating, even if it is a one man show (IMHO).


  • Secure: Security is constantly an afterthought which is a dangerous way to work in a build system. These systems are so complex and so critical to the company (they actually assemble and package the product) that we MUST make them secure by default. Sane secrets management, least access privileges, immutable systems, and forcing an audit trail all lead to a more secure and healthy environment.


  • Scalable: Build systems at successful companies only grow, they do not shrink. Teams are constantly adding more code, more tests, more jobs, and more artifacts. Normally this occurs in a polyglot environment leading to exponential growth of requirements on the machines that are actually doing the builds. It does not scale to have a single server with every single requirement installed. This pattern soon becomes unwieldly and hard to work with. We should containerize everything and separate the infrastructure from the system running on top of it to allow independent and customized scaling and maintenance.

If we can build a system that meets these requirements, we will get a lot of stuff for free as well including:


  • Easy disaster recovery
  • Portability to different service providers
  • Fun working environment
  • High levels of productivity
  • Profit!

An iterative approach

Each iteration will build a shippable increment on top of the iteration before it and eventually we will have a production Jenkins! Along the way we will learn a bunch of stuff, build some Docker containers, write a compose file, automate some configuration, get real groovy, and much, much more. I encourage you to stay tuned and work through solving this problem with me.

All code will be published to this blog’s git repo 3 so that you can verify your answers and fix inconsistencies if you get stuck along the way. The repo should be tagged to match up with this series.

Now that we have a general idea of how we want our system to behave and how we will build it out, let’s dig into the architectural concerns.

Next Post: Architecting a Jenkins + Docker CI System