Making HTTP GET and POST Requests to Docker Hub in a Jenkinsfile

Reading time ~18 minutes

NOTE: If you intend on using techniques such as these and allowing such wide open functionality in Jenkins, I recommend that you run your entire Jenkins build system without outbound internet access by default. Allow access from the build system network segment only to approved endpoints while dropping the rest of the traffic. This will allow you to use potentially dangerous, but extremely powerful scripts while maintaining a high level of security in the system.

API Calls

Jenkinsfile calling Docker API

Making HTTP calls in a Jenkinsfile can prove tricky as everything has to remain serializable else Jenkins cannot keep track of the state when needing to restart a pipeline. If you’ve ever received Jenkins’ java.io.NotSerializableException error in your console out, you know what I mean. There are not too many clear-cut examples of remote API calls from a Jenkinsfile so I’ve created an example Jenkinsfile that will talk to the Docker Hub API using GET and POST to perform authenticated API calls.

Explicit Versioning

The Docker image build process leaves a good amount to be desired when it comes to versioning. Most workflows depend on the :latest tag which is very ambiguous and can lead to problems being swallowed within your build system. In order to maintain a higher level of determinism and auditability, you may consider creating your own versioning scheme for Docker images.

For instance, a version of 2017.2.1929 #<year>-<week>-<build #> can express much more information than a simple latest. Having this information available for audits or tracking down when a failure was introduced can be invaluable, but there is no built-in way to do Docker versioning in Jenkins. One must rely on an external system (such as Docker Hub or their internal registry) to keep track of versions and use this system of record when promoting builds.


Versioning Scheme


This versioning scheme we are using is not based on Semver1, but it does encode within it the information we need to keep versions in lock and also will always increase in value. Even if the build number is reset, the date + week will keep the versions from ever being lower that the day previously. Version your artifacts however works for your release, but please make sure of these two things:

  • The version string never duplicates
  • The version number never decreases

Interacting with the Docker Hub API in a Jenkinsfile

For this example we are going to connect to the Docker Hub REST API in order to retrieve some tags and promote a build to RC. This type of workflow would be implemented in a release job in which a previously built Docker image is being promoted to a release candidate. The steps we take in the Jenkinsfile are:

Docker Registry

  • Provision a node
    • Stage 1
      • Make an HTTP POST request to the Docker Hub to get an auth token
      • Use the token to fetch the list of tags on an image
      • Filter through those tags to find a tag for the given build #
    • Stage 2
      • Promote (pull, tag, and push) the tag found previously as ${version}-rc
      • Push that tag to latest to make it generally available

This is a fairly complex looking Jenkinsfile as it stands, but these functions can be pulled out into a shared library2 to simplify the Jenkinsfile. We’ll talk about that in another post.


Jenkinsfile

#!groovy
/*
 NOTE: This Jenkinsfile has the following pre-requisites:
   - SECRET (id: docker-hub-user-pass): Username / Password secret containing your
     Docker Hub username and password.
   - ENVIRONMENT: Docker commands should work meaning DOCKER_HOST is set or there
     is access to the socket.
*/
import groovy.json.JsonSlurperClassic; // Required for parseJSON()

// These vars would most likely be set as parameters
imageName = "technolog/serviceone"
build = "103"

// Begin our Scripted Pipeline definition by provisioning a node
node() {

    // First stage sets up version info
    stage('Get Docker Tag from Build Number') {

        // Expose our user/pass credential as vars
        withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
            // Generate our auth token
            token = getAuthTokenDockerHub(user, pass)
        }

        // Use our auth token to get the tag
        tag = getTagFromDockerHub(imageName, build, token)
    }

    // Example second stage tags version as -release and pushes to latest
    stage('Promote build to RC') {
        // Enclose in try/catch for cleanup
        try {
            // Define our versions
            def versionImg = "${imageName}:${tag}"
            def latestImg = "${imageName}:latest"

            // Login with our Docker credentials
            withCredentials([usernamePassword(credentialsId: 'docker-hub-user-pass', passwordVariable: 'pass', usernameVariable: 'user')]) {
                sh "docker login -u${user} -p${pass}"
            }

            // Pull, tag, + push the RC
            sh "docker pull ${versionImg}"
            sh "docker tag ${versionImg} ${versionImg}-rc"
            sh "docker push ${versionImg}-rc"

            // Push the RC to latest as well
            sh "docker tag ${versionImg} ${latestImg}"
            sh "docker push ${latestImg}"
        } catch(err) {
            // Display errors and set status to failure
            echo "FAILURE: Caught error: ${err}"
            currentBuild.result = "FAILURE"
        } finally {
            // Finally perform cleanup
            sh 'docker system prune -af'
        }
    }
}

// NOTE: Everything below here could be put into a shared library

// GET Example
// Get a tag from Docker Hub for a given build number
def getTagFromDockerHub(imgName, build, authToken) {

    // Generate our URL. Auth is required for private repos
    def url = new URL("https://hub.docker.com/v2/repositories/${imgName}/tags")
    def parsedJSON = parseJSON(url.getText(requestProperties:["Authorization":"JWT ${authToken}"]))

    // We want to find the tag associated with a build
    // EX: 2017.2.103 or 2016.33.23945
    def regexp = "^\\d{4}.\\d{1,2}.${build}\$"

    // Iterate over the tags and return the one we want
    for (result in parsedJSON.results) {
        if (result.name.findAll(regexp)) {
            return result.name
        }
    }
}

// POST Example
// Get an Authentication token from Docker Hub
def getAuthTokenDockerHub(user, pass) {

    // Define our URL and make the connection
    def url = new URL("https://hub.docker.com/v2/users/login/")
    def conn = url.openConnection()
    // Set the connection verb and headers
    conn.setRequestMethod("POST")
    conn.setRequestProperty("Content-Type", "application/json")
    // Required to send the request body of our POST
    conn.doOutput = true

    // Create our JSON Authentication string
    def authString = "{\"username\": \"${user}\", \"password\": \"${pass}\"}"

    // Send our request
    def writer = new OutputStreamWriter(conn.outputStream)
    writer.write(authString)
    writer.flush()
    writer.close()
    conn.connect()

    // Parse and return the token
    def result = parseJSON(conn.content.text)
    return result.token

}

// Contain our JsonSlurper in a function to maintain CPS
def parseJSON(json) {
    return new groovy.json.JsonSlurperClassic().parseText(json)
}

Script Security

Due to the nature of this type of script, there is definitely a lot of trust assumed when allowing something like this to run. If you follow the process we are doing in Modern Jenkins nothing is getting into the build system without peer review and nobody but administrators have access to run scripts like this. With the environment locked down, it can be safe to use something of this nature.

Security Alert

Jenkins has two ways in which Jenkinsfiles (and Groovy in general) can be run: sandboxed or un-sandboxed. After reading Do not disable the Groovy Sandbox by rtyler (@agentdero on Twitter), I will never disable sandbox again. What we are going to do instead is whitelist all of the required signatures automatically with Groovy. The script we are going to use is adapted from my friend Brandon Fryslie and will basically pre-authorize all of the required methods that the pipeline will use to make the API calls.

Pre-authorizing Jenkins Signatures with Groovy


URL: http://localhost:8080/script

import org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval

println("INFO: Whitelisting requirements for Jenkinsfile API Calls")

// Create a list of the required signatures
def requiredSigs = [
    'method groovy.json.JsonSlurperClassic parseText java.lang.String',
    'method java.io.Flushable flush',
    'method java.io.Writer write java.lang.String',
    'method java.lang.AutoCloseable close',
    'method java.net.HttpURLConnection setRequestMethod java.lang.String',
    'method java.net.URL openConnection',
    'method java.net.URLConnection connect',
    'method java.net.URLConnection getContent',
    'method java.net.URLConnection getOutputStream',
    'method java.net.URLConnection setDoOutput boolean',
    'method java.net.URLConnection setRequestProperty java.lang.String java.lang.String',
    'new groovy.json.JsonSlurperClassic',
    'new java.io.OutputStreamWriter java.io.OutputStream',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods findAll java.lang.String java.lang.String',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.io.InputStream',
    'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.net.URL java.util.Map',

    // Signatures already approved which may have introduced a security vulnerability (recommend clearing):
    'method java.net.URL openConnection',
]

// Get a handle on our approval object
approver = ScriptApproval.get()

// Aprove each of them
requiredSigs.each {
    approver.approveSignature(it)
}

println("INFO: Jenkinsfile API calls signatures approved")

After running this script, you can browse to Manage Jenkins -> In-process Script Approval and see that there is a list of pre-whitelisted signatures that will allow our Jenkinsfile to make the necessary calls to interact with the Docker Hub API. You’ll notice there is one method in there that they mark in red as you can see the potential security issues with it. java.net.URL openConnection can be an extremely dangerous method to allow in an unrestricted environment so be careful and make sure you have things locked down in other ways.

Update from Matt

Matt has been working on big art recently, including Double Diamond and Moonrock Mountain. They are both large-scale sculptures that incorporate everything he has learned throughout his career. Continue reading