NOTE: If you intend on using techniques such as these and allowing such wide open functionality in Jenkins, I recommend that you run your entire Jenkins build system without outbound internet access by default. Allow access from the build system network segment only to approved endpoints while dropping the rest of the traffic. This will allow you to use potentially dangerous, but extremely powerful scripts while maintaining a high level of security in the system.
API Calls
Making HTTP calls in a Jenkinsfile can prove tricky as everything has to remain
serializable else Jenkins cannot keep track of the state when needing to restart
a pipeline. If you’ve ever received Jenkins’ java.io.NotSerializableException
error in your console out, you know what I mean. There are not too many clear-cut
examples of remote API calls from a Jenkinsfile so I’ve created an example
Jenkinsfile that will talk to the Docker Hub API using GET
and POST
to
perform authenticated API calls.
Explicit Versioning
The Docker image build process leaves a good amount to be desired when it comes
to versioning. Most workflows depend on the :latest
tag which is very
ambiguous and can lead to problems being swallowed within your build system. In
order to maintain a higher level of determinism and auditability, you may
consider creating your own versioning scheme for Docker images.
For instance, a version of 2017.2.1929 #<year>-<week>-<build #>
can express
much more information than a simple latest
. Having this information available
for audits or tracking down when a failure was introduced can be invaluable, but
there is no built-in way to do Docker versioning in Jenkins. One must rely
on an external system (such as Docker Hub or their internal registry) to keep
track of versions and use this system of record when promoting builds.
This versioning scheme we are using is not based on Semver1, but it does encode within it the information we need to keep versions in lock and also will always increase in value. Even if the build number is reset, the date + week will keep the versions from ever being lower that the day previously. Version your artifacts however works for your release, but please make sure of these two things:
- The version string never duplicates
- The version number never decreases
Interacting with the Docker Hub API in a Jenkinsfile
For this example we are going to connect to the Docker Hub REST API in order to retrieve some tags and promote a build to RC. This type of workflow would be implemented in a release job in which a previously built Docker image is being promoted to a release candidate. The steps we take in the Jenkinsfile are:
- Provision a node
- Stage 1
- Make an HTTP POST request to the Docker Hub to get an auth token
- Use the token to fetch the list of tags on an image
- Filter through those tags to find a tag for the given build #
- Stage 2
- Promote (pull, tag, and push) the tag found previously as ${version}-rc
- Push that tag to
latest
to make it generally available
- Stage 1
This is a fairly complex looking Jenkinsfile as it stands, but these functions can be pulled out into a shared library2 to simplify the Jenkinsfile. We’ll talk about that in another post.
Jenkinsfile
#!groovy
/*
NOTE: This Jenkinsfile has the following pre-requisites:
- SECRET (id: docker-hub-user-pass): Username / Password secret containing your
Docker Hub username and password.
- ENVIRONMENT: Docker commands should work meaning DOCKER_HOST is set or there
is access to the socket.
*/
import groovy.json.JsonSlurperClassic; // Required for parseJSON()
// These vars would most likely be set as parameters
imageName = "technolog/serviceone"
build = "103"
// Begin our Scripted Pipeline definition by provisioning a node
node() {
// First stage sets up version info
stage('Get Docker Tag from Build Number') {
// Expose our user/pass credential as vars
withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
// Generate our auth token
token = getAuthTokenDockerHub(user, pass)
}
// Use our auth token to get the tag
tag = getTagFromDockerHub(imageName, build, token)
}
// Example second stage tags version as -release and pushes to latest
stage('Promote build to RC') {
// Enclose in try/catch for cleanup
try {
// Define our versions
def versionImg = "${imageName}:${tag}"
def latestImg = "${imageName}:latest"
// Login with our Docker credentials
withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
sh "docker login -u${user} -p${pass}"
}
// Pull, tag, + push the RC
sh "docker pull ${versionImg}"
sh "docker tag ${versionImg} ${versionImg}-rc"
sh "docker push ${versionImg}-rc"
// Push the RC to latest as well
sh "docker tag ${versionImg} ${latestImg}"
sh "docker push ${latestImg}"
} catch(err) {
// Display errors and set status to failure
echo "FAILURE: Caught error: ${err}"
currentBuild.result = "FAILURE"
} finally {
// Finally perform cleanup
sh 'docker system prune -af'
}
}
}
// NOTE: Everything below here could be put into a shared library
// GET Example
// Get a tag from Docker Hub for a given build number
def getTagFromDockerHub(imgName, build, authToken) {
// Generate our URL. Auth is required for private repos
def url = new URL("https://hub.docker.com/v2/repositories/${imgName}/tags")
def parsedJSON = parseJSON(url.getText(requestProperties:["Authorization":"JWT ${authToken}"]))
// We want to find the tag associated with a build
// EX: 2017.2.103 or 2016.33.23945
def regexp = "^\\d{4}.\\d{1,2}.${build}\$"
// Iterate over the tags and return the one we want
for (result in parsedJSON.results) {
if (result.name.findAll(regexp)) {
return result.name
}
}
}
// POST Example
// Get an Authentication token from Docker Hub
def getAuthTokenDockerHub(user, pass) {
// Define our URL and make the connection
def url = new URL("https://hub.docker.com/v2/users/login/")
def conn = url.openConnection()
// Set the connection verb and headers
conn.setRequestMethod("POST")
conn.setRequestProperty("Content-Type", "application/json")
// Required to send the request body of our POST
conn.doOutput = true
// Create our JSON Authentication string
def authString = "{\"username\": \"${user}\", \"password\": \"${pass}\"}"
// Send our request
def writer = new OutputStreamWriter(conn.outputStream)
writer.write(authString)
writer.flush()
writer.close()
conn.connect()
// Parse and return the token
def result = parseJSON(conn.content.text)
return result.token
}
// Contain our JsonSlurper in a function to maintain CPS
def parseJSON(json) {
return new groovy.json.JsonSlurperClassic().parseText(json)
}
Script Security
Due to the nature of this type of script, there is definitely a lot of trust assumed when allowing something like this to run. If you follow the process we are doing in Modern Jenkins nothing is getting into the build system without peer review and nobody but administrators have access to run scripts like this. With the environment locked down, it can be safe to use something of this nature.
Jenkins has two ways in which Jenkinsfiles (and Groovy in general) can be run: sandboxed or un-sandboxed. After reading Do not disable the Groovy Sandbox by rtyler (@agentdero on Twitter), I will never disable sandbox again. What we are going to do instead is whitelist all of the required signatures automatically with Groovy. The script we are going to use is adapted from my friend Brandon Fryslie and will basically pre-authorize all of the required methods that the pipeline will use to make the API calls.
Pre-authorizing Jenkins Signatures with Groovy
URL: http://localhost:8080/script
import org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval
println("INFO: Whitelisting requirements for Jenkinsfile API Calls")
// Create a list of the required signatures
def requiredSigs = [
'method groovy.json.JsonSlurperClassic parseText java.lang.String',
'method java.io.Flushable flush',
'method java.io.Writer write java.lang.String',
'method java.lang.AutoCloseable close',
'method java.net.HttpURLConnection setRequestMethod java.lang.String',
'method java.net.URL openConnection',
'method java.net.URLConnection connect',
'method java.net.URLConnection getContent',
'method java.net.URLConnection getOutputStream',
'method java.net.URLConnection setDoOutput boolean',
'method java.net.URLConnection setRequestProperty java.lang.String java.lang.String',
'new groovy.json.JsonSlurperClassic',
'new java.io.OutputStreamWriter java.io.OutputStream',
'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods findAll java.lang.String java.lang.String',
'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.io.InputStream',
'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.net.URL java.util.Map',
// Signatures already approved which may have introduced a security vulnerability (recommend clearing):
'method java.net.URL openConnection',
]
// Get a handle on our approval object
approver = ScriptApproval.get()
// Aprove each of them
requiredSigs.each {
approver.approveSignature(it)
}
println("INFO: Jenkinsfile API calls signatures approved")
After running this script, you can browse to
Manage Jenkins -> In-process Script Approval
and see that there is a list of pre-whitelisted signatures that will allow our
Jenkinsfile to make the necessary calls to interact with the Docker Hub API.
You’ll notice there is one method in there that they mark in red as you can see
the potential security issues with it. java.net.URL openConnection
can be an
extremely dangerous method to allow in an unrestricted environment so be careful
and make sure you have things locked down in other ways.