NOTE: If you intend on using techniques such as these and allowing such wide
open functionality in Jenkins, I recommend that you run your entire Jenkins
build system without outbound internet access by default. Allow access from the
build system network segment only to approved endpoints while dropping the rest
of the traffic. This will allow you to use potentially dangerous, but extremely
powerful scripts while maintaining a high level of security in the system.
API Calls
Making HTTP calls in a Jenkinsfile can prove tricky as everything has to remain
serializable else Jenkins cannot keep track of the state when needing to restart
a pipeline. If you’ve ever received Jenkins’ java.io.NotSerializableException
error in your console out, you know what I mean. There are not too many clear-cut
examples of remote API calls from a Jenkinsfile so I’ve created an example
Jenkinsfile that will talk to the Docker Hub API using GET and POST to
perform authenticated API calls.
Explicit Versioning
The Docker image build process leaves a good amount to be desired when it comes
to versioning. Most workflows depend on the :latest tag which is very
ambiguous and can lead to problems being swallowed within your build system. In
order to maintain a higher level of determinism and auditability, you may
consider creating your own versioning scheme for Docker images.
For instance, a version of 2017.2.1929 #<year>-<week>-<build #> can express
much more information than a simple latest. Having this information available
for audits or tracking down when a failure was introduced can be invaluable, but
there is no built-in way to do Docker versioning in Jenkins. One must rely
on an external system (such as Docker Hub or their internal registry) to keep
track of versions and use this system of record when promoting builds.
This versioning scheme we are using is not based on Semver1, but it does
encode within it the information we need to keep versions in lock and also
will always increase in value. Even if the build number is reset, the date +
week will keep the versions from ever being lower that the day previously.
Version your artifacts however works for your release, but please make sure of
these two things:
The version string never duplicates
The version number never decreases
Interacting with the Docker Hub API in a Jenkinsfile
For this example we are going to connect to the Docker Hub REST API in order to
retrieve some tags and promote a build to RC. This type of workflow would be
implemented in a release job in which a previously built Docker image is being
promoted to a release candidate. The steps we take in the Jenkinsfile are:
Provision a node
Stage 1
Make an HTTP POST request to the Docker Hub to get an auth token
Use the token to fetch the list of tags on an image
Filter through those tags to find a tag for the given build #
Stage 2
Promote (pull, tag, and push) the tag found previously as ${version}-rc
Push that tag to latest to make it generally available
This is a fairly complex looking Jenkinsfile as it stands, but these functions
can be pulled out into a shared library2 to simplify the Jenkinsfile. We’ll
talk about that in another post.
Jenkinsfile
Script Security
Due to the nature of this type of script, there is definitely a lot of trust
assumed when allowing something like this to run. If you follow the process we
are doing in Modern Jenkins
nothing is getting into the build system without peer review and nobody but
administrators have access to run scripts like this. With the environment locked
down, it can be safe to use something of this nature.
Jenkins has two ways in which Jenkinsfiles (and Groovy in general) can be run:
sandboxed or un-sandboxed. After reading Do not disable the Groovy Sandbox
by rtyler (@agentdero on Twitter),
I will never disable sandbox again. What we are going to do instead is
whitelist all of the required signatures automatically with Groovy. The script
we are going to use is adapted from my friend Brandon Fryslie
and will basically pre-authorize all of the required methods that the pipeline
will use to make the API calls.
Pre-authorizing Jenkins Signatures with Groovy
URL: http://localhost:8080/script
After running this script, you can browse to
Manage Jenkins -> In-process Script Approval
and see that there is a list of pre-whitelisted signatures that will allow our
Jenkinsfile to make the necessary calls to interact with the Docker Hub API.
You’ll notice there is one method in there that they mark in red as you can see
the potential security issues with it. java.net.URL openConnection can be an
extremely dangerous method to allow in an unrestricted environment so be careful
and make sure you have things locked down in other ways.
Matt has been working on big art recently, including Double Diamond and Moonrock Mountain. They are both large-scale sculptures that incorporate everything he has learned throughout his career. Continue reading