Modern Jenkins Unit 3 / Part 2: Configure Jenkins URL

Configuring the Jenkins URL

Configure Jenkins URL with Groovy

Currently it just so happens that the Jenkins URL in the management console does have the proper config as it is set to http://localhost:8080. This is merely a coincidence that the default value and the current address match though. Once we start moving this thing around, it will be very important that it is set to the right value else we’ll have all kinds of strange issues. Since it will definitely have to be configured, let’s start here. It doesn’t hurt that it is a fairly simple example of configuring Jenkins with an environment variable passed by Docker Compose.

Passing from Docker Compose

This is a setting that will change from environment to environment and so I think the best place to set it is in the compose file. Let’s create a variable in there that we can read it in during init and make the right configuration. Edit the compose file and create a JENKINS_UI_URL env variable as well as volumes for the configs themselves.


deploy/master/docker-compose.yml

#---
## deploy/master/docker-compose.yml
## Define the version of the compose file we're using
#version: '3.3'
#
## Define our services
#services:
#  # Jenkins master's configuration
#  master:
#    image: modernjenkins/jenkins-master
#    ports:
#      - "8080:8080"
     environment:
        - JENKINS_UI_URL=http://localhost:8080
#    volumes:
#      - plugins:/usr/share/jenkins/ref/plugins
#      - warfile:/usr/share/jenkins/ref/warfile
       - groovy:/var/jenkins_home/init.groovy.d
#
#  # Jenkins plugins' configuration
#  plugins:
#    image: modernjenkins/jenkins-plugins
#    volumes:
#      - plugins:/usr/share/jenkins/ref/plugins
#      - warfile:/usr/share/jenkins/ref/warfile
       - groovy:/usr/share/jenkins/ref/init.groovy.d
#
## Define named volumes. These are what we use to share the data from one
## container to another, thereby making our jenkins.war and plugins available
#volumes:
#  plugins:
#  warfile:
   groovy:
#

Now that it is set, we should be able to write a groovy init script to read it. But first, we will have to restart Jenkins to pick up the new environment:


PWD: ~/code/modern-jenkins

cd deploy/master
docker-compose down -v
docker-compose up -d

# Confirm that the variable is there:
docker inspect master_master_1 | grep JENKINS_UI

# Should output
# "JENKINS_UI_URL=http://localhost:8080",

With the environment variable now being available to us, we can use the script console at localhost to develop our script that will set the URL of our Jenkins instance.


URL: http://localhost:8080/script

images/jenkins-plugins/files/init.groovy.d/01-cofigure-jenkins-url.groovy

import jenkins.model.*

// Read the environment variable
url = System.env.JENKINS_UI_URL

// Get the config from our running instance
urlConfig = JenkinsLocationConfiguration.get()

// Set the config to be the value of the env var
urlConfig.setUrl(url)

// Save the configuration
urlConfig.save()

// Print the results
println("Jenkins URL Set to " + url)

This should output:

URL Output

Deploying our Groovy Jenkins Configs

We now have a working script that will manage the URL of Jenkins in any environment that we would be deploying into simply by setting a variable in the compose file. Now we need to make this available to the master so that it can be executed on startup. To do that, we will add a director to the plugins image and then mount it into the master in a similar way to how the war and plugins work.


PWD: ~/code/modern-jenkins/

cd images/jenkins-plugins
mkdir -p files/init.groovy.d/

# Add the file above to this directory as
# files/init.groovy.d/01-cofigure-jenkins-url.groovy

Add the full directory (instead of the individual scripts) of Groovy configuration to the Docker image and then export it as a volume.


images/jenkins-plugins/Dockerfile

...
#  # Install our base set of plugins and their dependencies that are listed in
#  # plugins.txt
#  ADD files/plugins.txt /tmp/plugins-main.txt
#  RUN install-plugins.sh `cat /tmp/plugins-main.txt`
#
   # Add our groovy init files
   ADD files/init.groovy.d /usr/share/jenkins/ref/init.groovy.d
#
...

#  # Export our war and plugin set as volumes
#  VOLUME /usr/share/jenkins/ref/plugins
#  VOLUME /usr/share/jenkins/ref/warfile
   VOLUME /usr/share/jenkins/ref/init.groovy.d
#
#  ...

We can now rebuild the image and pick up our new config that should (hopefully) configure the URL of our Jenkins on boot!


PWD: ~/code/modern-jenkins

./images/jenkins-plugins/build.sh

cd deploy/master/
docker-compose down -v
docker system prune -f
docker-compose up -d
docker-compose logs -f

Ok you should now see that the URL in the Jenkins management console is set to http://localhost:8080! Ahhhh, it was like that before? Hmm. Ok well then let’s break it to confirm it is working. Modify the value in the compose file to something different and restart Jenkins:


PWD: ~/code/modern-jenkins/deploy/master

# Change the JENKINS_UI_URL to something different in docker-compose.yml
perl -pi -e 's/JENKINS_UI_URL=.*/JENKINS_UI_URL=http:\/\/derpyhooves/g' docker-compose.yml

# Restart the stack
docker-compose down -v
docker-compose up -d
docker-compos logs

Homer Fatfinger

Did that work? A typo you say? I can’t imagine it. I’ve typed docker-compose over 1000 times today, it’s not possible for me to misspell it. In addition, if you notice the difference between this set of commands and the one earlier, we seem to be drifting to chaos. Let’s take note of that, but address it after we confirm that this current change is working as expected.

Gooood, it does work :) The URL in the management console has been updated to the new, wrong, custom value so we can confirm it works. Let’s commit our code now, but attend to that tiny little pile of tech debt we found (starting and stopping the system differently every time is definitely tech debt). NOTE: Don’t forget to set the JENKINS_UI_URL back to http://localhost:8080


PWD: ~/code/modern-jenkins/

git checkout -b feat-configure_jenkins_url
git add .
git commit -m "Configure the Jenkins URL with Groovy

This change adds an environment variable to the compose file
that sets the URL of the Jenkins instance upon boot. This is
done via the script added to init.groovy.d"

Squirrel!

Squirrel

So we’ve noticed something a bit stinky in our code as we’ve been going about our business. This happens very often in our work lives and attending to little tech debts like this is critical to having quality software. I certainly encourage everyone to leave the code better than they found it and to refactor things when they see something turning into a turd like object.

I also encourage you to make note of these things and take care of them after you are in a place to save game. Switching context between one problem and another can be very expensive mind and time wise so feel free to take a note, create a ticket or something, then finish what you are doing (unless there is an actual issue that needs to be addressed before your code will work). When you submit your PR for review, then jump on that Jira and refactor your heart away.

We need to do everything, but we can only do one thing at a time. Try to be aware of time management.

Let’s get that squirrel

What we noticed was that we were beginning to type the command differently every time we did it. That seems like it should be replaced by a shell script. Let’s create a start script so that this thing starts consistently every time:


deploy/master/start.sh

#!/bin/bash -el

echo "INFO: (re)Starting Jenkins"
docker-compose down -v
docker-compose up -d

echo "INFO: Use the following command to watch the logs: "
echo "docker-compose logs -f"

Write that guy out to deploy/master/start.sh and make it executable with chmod +x deploy/master/start.sh and we’ve now got ourselves a script that will consistently restart our app.

Add that onto the branch with a good message and push, PR, merge, etc.. See, you’re getting the hang of it!

Next let’s get ready to handle some secrets. Shhhhhh!

The repo from this section can be found under the unit3-part2 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit3-part2

Next Post: Managing Jenkins secrets in GitHub with Transcrypt

Modern Jenkins Unit 3 / Part 1: Intro to the Jenkins Groovy Init System

Professor Frink Configuring Bender Image from Simpsons Mathematics

Configuring the Jenkins Master on Boot with Groovy

One of the greatest strengths of Jenkins is it’s ability to do almost anything. This comes from it’s extremely customizable nature. I think of it as a scheduler that can do any given task on any given trigger with enough configuration. What we are building specifically though is a system to build, test, and deploy our piece of custom software that most likely is at least a little bit different than anything else out there requiring the same tasks. We will need to use Jenkins’ ultra powerful customization toolkit, but do it in a way that strives to be:

  1. Deterministic: Given a set of inputs, there should be only one output. If that output is not as we expect, there is a problem that we should spend time fixing. ie: removing “flaky” tests that fail for “no reason”.

  2. Reliable: The system should have high availability to the users who depend on it. Having a system that is sometimes down and sometimes up does not encourage teams to inject automated builds into their workflows.

  3. Repeatable: This system should be able to be recreated without persistent data from the repo.

  4. Agile: The system should evolve to meet the needs of it’s consumers in a sustainable way. If one team’s jobs or configs are breaking another team’s pipelines, it is a good indication that it is time to split the monolith into two independent systems.

  5. Scalable: As the system becomes more popular, more people are going to utilize it. When this happens, it’s critical to be able to support the increased capacity in not only job runners, but also collaboration from more teams.

Luckily we can treat the code that configures the system in the same way we treat the code the builds and runs the system :)

Intro to the Jenkins init system

Jenkins has a not-much-talked about feature that I have yet to see much information on. It is the Jenkins Groovy Init system and really the only documentation I have been able to find are two articles on the Jenkins wiki: https://wiki.jenkins.io/display/JENKINS/Post-initialization+script


  Post-initialization script
  
  Created by Kohsuke Kawaguchi, last modified by Daniel Beck on Dec 10, 2015
  
  You can create a Groovy script file $JENKINS_HOME/init.groovy, or any .groovy
  file in the directory $JENKINS_HOME/init.groovy.d/, (See Configuring Jenkins
  upon start up for more info) to run some additional things right after Jenkins
  starts up. This script can access classes in Jenkins and all the plugins. So for
  example, you can write something like:
  import jenkins.model.*;
  
  // start in the state that doesn't do any build.
  Jenkins.instance.doQuietDown();
  
  
  Output is logged to the Jenkins log file. For Debian based users, this is
  /var/log/jenkins/jenkins.log

which points to this: https://wiki.jenkins.io/display/JENKINS/Configuring+Jenkins+upon+start+up


  Jenkins can execute initialization scripts written in Groovy if they are present
  during start up. See Groovy Hook Script for details. The hook name for this
  event is "init". Those executions happen at the very end of the initialization,
  and therefore this can be used to pre-configure Jenkins for a particular OEM
  situation. 
  
  While one can always write a plugin to participate in the initialization of
  Jenkins, this script-based approach can be useful as it doesn't require any
  compilation and packaging.

Not super impressive documentation considering how powerful this mechanism is. Using this init system you are able to configure any aspect of the master that you are able to using “Manage Jenkins”. This includes (but is not limited to):

  • The URL and name of this instance
  • Authentication and security settings
  • Secrets in the credential store
  • Global plugin configuration
  • Global tool configuration
  • Adding and removing nodes
  • Creation of jobs (though we’ll only use it to create one special job)

Groovy Configuration


Jenkins Groovy Script Console

The Groovy Script Console

Not only does it support configuring so much of the system, it has a direct REPL like environment to code in. This is called the “Script Console” (available at http://localhost:8080/script on our instance) and can be considered a shell into the Jenkins system. This shell has the same abilities of the init system’s shell so it makes for super quick and easy development of the scripts that we will use.

Jenkins Groovy Hello World

Let’s kill two stones with one bird. We will do a quick little Hello World that will introduce you to bot the syntax of groovy as well as how to use the script console.

  • Stand up your development Jenkins (cd deploy/master && docker-compose up -d)
  • Browse to the script console at http://localhost:8080/script
  • Enter the following into the box in front of you:

URL: http://localhost:8080/script

  import jenkins.model.*

  def jenkins = Jenkins.getInstance()
  jenkins.setSystemMessage("I'm Bender, baby! Oh god, please insert liquor!")
  // You can change the message if you please. I'm not at the office :)

  • Browse back to the main Jenkins interface
  • Check out the cool message for all the users of your system to see. I bet your boss will love it!

I'm Bender Baby!


Nothing too crazy, but this should give you a good idea of how we are going to configure our master to build our particular brand of software. Inside Old Mr. Jenkins is just a series of objects (I think of them as his organs) that we can grab and squeeze and modify to fit our needs. I hope Janky is ready to play “Operation”!

Next Post: Configure Jenkins URL with Groovy on boot

Modern Jenkins Unit 2 / Part 5: Starting Jenkins with Docker Compose

“The Good Ole’ Days”

Trollface

Back in aught eight when I was a kid, the way we deployed complex services was a 1000 line shell script that was neither idempotent nor checked into SCM. It just sat there at an http endpoint, ready for sudo | bashing (I guess normally sudo wasn’t an issue as we ran as root :P). If it needed a tweak, you could just ssh to the server, fire up pico, make the change, deploy your stuff, sit back and relax while wondering why the rest of the team is complaining about deploys not working. After all, it Works on My Machine :)

While I look back with fondness at the days of yore, I can only imagine it is the fresh Colorado air that is making me forget how crappy it is to have one deploy work and then literally the same thing 5 minutes later fails because someone was mucking with the script. So we’re not going to do that.

Docker Compose

Instead, we are going to use something called Docker Compose. Docker Compose is a project by Docker that was based on something called fig a long time ago. Unlike the rest of their toolkit, docker-compose is a Python application that uses YAML to describe a service or set of services. It allows you to define pretty much every aspect of how the services are run, what the networking and storage systems will look like, and to fine tune how your app will work via environment variables.

There is a ton of info out there on Docker Compose 1 so please do take a peek. For now, let’s roll forward into the unknown and create our first compose file.


deploy/master/docker-compose.yml

---
# deploy/master/docker-compose.yml
# Define the version of the compose file we're using
version: '3.3'

# Define our services
services:
  # Jenkins master's configuration
  master:
    image: modernjenkins/jenkins-master
    ports:
      - "8080:8080"
    volumes:
      - plugins:/usr/share/jenkins/ref/plugins
      - warfile:/usr/share/jenkins/ref/warfile

  # Jenkins plugins' configuration
  plugins:
    image: modernjenkins/jenkins-plugins
    volumes:
      - plugins:/usr/share/jenkins/ref/plugins
      - warfile:/usr/share/jenkins/ref/warfile

# Define named volumes. These are what we use to share the data from one
# container to another, thereby making our jenkins.war and plugins available
volumes:
  plugins:
  warfile:

A compose file is made up of a few sections as in the example above. Here the ones we’re using:

  • version 2: Define what version of compose file this is

  • services 3: This is where we list out all of the services that we need running. This example is fairly straightforward, but it is possible to include any service your app needs in this section. You’re basically describing the full system and it’s interactions.

  • volumes 4: This is where data storage is described. We’re using it to define two volumes, one for plugins and one for the warfile. Upon creating this volume, data from the container will be copied in. Since the first container does not have anything at that path, the data from the second container is what we get which is exactly what we want.

  • networks 5: Not used here, but a way to define all container networking.

This is a fairly simple example of a compose file so it should be fairly straightforward to understand. You may also notice that it’s very succinct and to the point while still being super readable. This is why I like Docker Compose. We can describe something extremely complex (not so much in this case) as an easy to read YAML file.

Test it out

Ok, here’ we go girls and boys. The big reveal. Our rocket-powered-motorcycle is fueled up and we’re ready to jump the Snake river!


PWD: ~/code/modern-jenkins

# Compose up
cd deploy/master
docker-compose up -d
docker-compose logs -f

The Jenkins app should be starting up now and once it says “Jenkins is fully up and running” you should be able to browse to the UI at http://localhost:8080 and bask in its Janky glory.

Now that we know how to start / stop it, we should add this to the documentation. It is important to keep these docs up to date so that anyone can jump in and start using it without having to do a massive amount of research. Let’s add this to the README:


deploy/README.md

# Deployment Scripts

Jenkins is deployed via Docker Compose. In order to run the service locally, use
the following commands:

```
# Get into the deploy directory
cd deploy/master

# Start the service as a daemon
docker-compose up -d

# View logs
docker-compose logs -f

# Stop Jenkins
docker-compose down -v

# Pull new images
docker-compose pull
```

WTF Matt, a 6 part blog series to replace “java -jar jenkins.war” and 6 clicks?

hahaha, well you got me there! JK. While java -jar jenkins.war and a few mouse clicks could get us to the same place, it would not have been wasting nearly enough bits :trollface:

Crazy like Fox News

Obviously there are two potential reasons why we did this:

  1. I am a crazy person
  2. I am a just-crazy-enough person

Luckily for you, the answer is the latter. If you’ll recall, the whole reason I’m writing this is because I’m tired of people showing me their ugly Jenkins and encouraging me to tell them how cute it is.

The problem with most of these monstrosities is not that they don’t build the software. If it didn’t do that it wouldn’t exist at all. The problem is that they are extremely hard, time consuming, dangerous, and unfun to work on.

Spaghetti Code

That’s fine for something that never changes, but as it turns out we’re making things for the internet which is growing and changing constantly meaning that we constantly need to change and adapt in order to move forward. This applies very much so to our build system. It is a system that eventually everyone in the company begins to rely on, from C levels that need profits to PMs and POs who need features, to Engineers who need to do the actual work.

When a CI system turns into a bowl of spaghetti each little necessary change becomes a nerve-racking-afterhours-signed-out-of-slack maintenance that gives us all PTSD after the 30th time it goes south. What we are doing here is implementing a semi-rigid structure for our system to basically manage change effectively while still moving fast.

Jenkins Oops

Let’s walk through some of the common Crappy Times at Jenkins High:

Cincinnati Time Waste

  • A plugin with a broken dependency: Instead of finding out after checking the ‘restart Jenkins when done’ that a plugin can’t fully resolve it’s dependencies, we will see it when we try to build the Docker image. It is still non-optimal that it’s happening, but it is not a prod outage and builds are still running, preventing a Cincinnati Time Waste tournament.

  • Rolling credentials for the CI Git user: In the old days, this required a ton of coordination in addition to an outage. We have not yet showed it, but when your secrets are tied to the container we are able to modify all the required credentials, roll the system, and get back at it.

  • A job that broke for “no reason”: It’s always unpleasant to be surprised by a job that is just no longer going green. When we version all of our jobs, plugins, and master configuration, bisecting what caused a failure (or behavior change) becomes much simpler. We just go back to the point in which the job was running and diff the environment to what is running today. Since we’re able to run everything locally it should be a fairly straightforward process to replicate the problem on your laptop and lower your MTTR.

All of these problems we are talking about are still going to occur, but what we’re doing is pushing the problems down to build time from runtime. We want to find these issues in advance where they are not causing outages. We want to be able to treat our pipelines, configuration, and infrastructure as code to avoid the bowl of spaghetti that is fragile and unkown in nature. The teams should not be called “10ft Pole” (my old team) that help with the build system, they should be called “Golden Retriever Puppies” because everyone wants to play with us.

Golden Retriever Puppies

In conclusion

In conclusion, I hope you are able to see how the beginnings of our system are going to lend themselves to being a fully scalable solution that can scale to hundreds of builds, thousands of developers, and at least 10s of different companies you’re going to work at :)

If you don’t see it quite yet then you’re going to have to trust me that we are indeed doing this work for something and not for nothing. Anyways, no skin off of my nose if you don’t. Just keep typing code monkey.

In the next unit of this series we will begin configuring Jenkins. This will allow you to begin making Jenkins do the stuff you need it to do. Stay tuned for Unit 3 of Modern Jenkins: Programmatically Configuring Jenkins for Immutability with Groovy.

The repo from this section can be found under the unit2-part5 tag here: https://github.com/technolo-g/modern-jenkins/tree/unit2-part5

Next Post: The Jenkins Groovy init system (init.groovy.d)