In my earlier posts, I discussed using Gradle Rules to automate devops scripts. I’ve been working on a automated deployment pipeline, including a "rolling" upgrade of servers behind an Apache HTTPD Load Balancer. (See Gradle Rules! and Load Balancing).

Simple, with the right tool

However, I made a mistake that no experienced developer should ever make when I said:

Each of these are relatively simple scripting tasks.

I had not gotten to the bit about "Put target server in 'drain' mode" using the Balancer Manager. I tried using Groovy and java.util.URL to scrape the balancer-manager page with a pile of regex and POST a form to change a given worker’s drain mode. Long story short: balancer-manager is fussy about it’s 'nonce' value and quitely ignores any errors in the form. I couldn’t work out why it wouldn’t accept my POST request. So I did the rational thing and gave up.

Yesterday, a blog post showed up in my feeds that had the words "Geb" and "Automation". I didn’t read it, I just slapped my forehead, shouted "Doh!" and crapped out this script:

#!/usr/bin/env groovy
package devops

//Uncomment these lines to run it as a script, otherwise compile it with Gradle.
//@Grapes([
//        @Grab("org.gebish:geb-core:0.10.0"),
//        @Grab("org.seleniumhq.selenium:selenium-support:2.23.1"),
//        @Grab("org.seleniumhq.selenium:selenium-htmlunit-driver:2.44.0")
//
//])
import geb.Browser

def balancerURL = 'https://example.com/balancer-manager'
def cli = new CliBuilder(usage: 'balancermanager [options] worker')
cli.d(args: 1, "drain")
cli.b(args: 1, "balancer URL")
cli.h('help')
def options = cli.parse(args)

if (options.b) {
    balancerURL = options.b
}

def drain = false
if (options.d) {
    drain = options.d.toBoolean()
}
def worker = options.arguments().head()

Browser.drive {

    go balancerURL
    if (driver.currentUrl != balancerURL) {
        println "error: balancer manager at $balancerURL not found or unavailable"
        System.exit(1)
    }

    def link = $('a').find { it.toString().contains worker }
    if (!link) {
        println "error: $worker not found in balancer at $balancerURL"
        System.exit(1)
    }

    link.click()

    def drainRadio = $("input", name: 'w_status_N', value: drain ? '1' : '0')

    drainRadio.click()

    $('input', type: 'submit').click()

    def w = $("tr").find { it.text().contains worker}
    def status =  w.children().collect { it.text() }.join(' ')
    println "$worker set to drain mode: $drain  (status: $status)"

}

We’ve used Geb for years for integration/functional tests. But it’s also a perfect tool for this type of task. Geb with the htmlunit WebDriver is a plenty good web browser to interact with the balancer-manager.

With this script, I can do:

./balancermanager -d true MyServer01   # turn on drain mode
./balancermanager -d false MyServer01  # turn off drain mode

Gradle Rule for Draining

But I don’t really want a script, I want a Gradle task to change a given worker’s drain mode. So this rule goes into my devops build.gradle:

tasks.addRule("Pattern: drain{On|Off}For<server>: set drain mode on|off balancer for <server>") { String taskName ->

    def matcher = (taskName =~ /drain(.*?)For(.*)/)
    if (matcher.matches()) {
        def (drainMode, server) = matcher[0][1..2]*.toLowerCase()

        drainMode = drainMode == 'on' ? true : false
        task(type: JavaExec, taskName) {
            description "${drainMode ? 'Enable' : 'Disable'} drain mode for $server"
            main = 'devops.balancermanager'
            classpath = sourceSets.main.runtimeClasspath
            args "-d $drainMode"
            args "$server"
        }
    }
}

Now I can do:

./gradlew drainOnForMyServer01
./gradlew drainOffForMyServer01

Gradle Rule for [Un]Deploy

We use Cargo with the Gradle Cargo Plugin to handle remote Tomcat deployments. The plugin is quite flexible and allows you to create your own tasks, so it can be made into a rule:

tasks.addRule("Pattern: [un]deploy<context>On<server>: undeploy or deploy /<context> on <server>") { String taskName ->

    def matcher = (taskName =~ /(undeploy|deploy)(.*?)On(.*)/)
    if (matcher.matches()) {

        def (action, targetContext, server) = matcher[0][1..3]
        def cargoTaskType = action == 'deploy' ? CargoDeployRemote : SafeUndeployRemote

        task "$taskName"(type: cargoTaskType) {
            description "$action /$context on Tomcat $server"
            hostname = server.toLowerCase()

            if (deployables.size() != 1) {
                throw new GradleException("cargo rule supports exactly one deployable. found ${deployables.size()}")
            }
            deployables.first().context = '/' + targetContext.toLowerCase()
        }
    }
}

class SafeUndeployRemote extends CargoUndeployRemote {

    @Override
    void runAction() {
        try {
            super.runAction()
        } catch (e) { println "undeploy failed: $e.message"}
    }
}
Note
I’m only showing the bits of Cargo for creating the rule. It assumes a normal cargo {} configuration block for the other Cargo parameters (manager username/password, port, etc). Those are normal gradle or project properties. I’m not trying to replace all properties. Just the bits that are different for each environment.

Notice that the 'undeploy' is a special case. The default Cargo undeploy task will fail the build if it can’t undeploy. This includes the case where the context does not exist. In my case, I’m happy the thing doesn’t exist. So I subclass the task and eat the exception.

Tying it Together

Now I have all the pieces to perform my style of "rolling upgrade" on a single server. I can do this:

./gradlew drainOnServer-01 waitForMyAppSessionsOnServer-01 undeployMyAppOnServer-01 deployMyAppOnServer-01 drainOffServer-01

Cool! Right? Wrong…​ I’ve had to repeat the server name five times and the context three. Not cool. Once more, we need some Rule magic to make it DRY:

tasks.addRule("Pattern: rollingUpgrade<context>On<server>: 'rolling' upgrade of <context> on <server>") { String taskName ->

    def matcher = (taskName =~ /rollingUpgrade(.*?)On(.*)/)
    if (matcher.matches()) {

        def (context, server) = matcher[0][1..2]

        // rolling tasks in the order they must run
        def rollingTasks = [ "drainOnFor$server",
                     "waitFor${context}SessionsOn$server",
                     "undeploy${context}On$server",
                     "deploy${context}On$server",
                     "drainOffFor$server" ]

        task "$taskName" {
            dependsOn rollingTasks

            rollingTasks.tail().inject(rollingTasks.head()) { a, b ->
                tasks[b].mustRunAfter a
                b
            }
        } << {
            println "Completed rolling upgrade of $context on $server"
        }
    }
}

Now, I can just do this in my Bamboo deployment task:

./gradlew rollingUpgradeMyAppOnServer-01
./gradlew rollingUpgradeMyAppOnServer-02
./gradlew rollingUpgradeMyAppOnServer-03

Now that’s pretty cool.

If you’re not familiar with the difference between dependsOn and mustRunAfter, it bares some discussion. mustRunAfter specifies order of task two tasks if both are scheduled to run. But does not require they both be run.

If I use dependsOn, like:

taskA.dependsOn taskB

Then I can’t run taskA without also running taskB. But if I use this instead:

taskA.mustRunAfter taskB

I can run taskA without taskB. But if I run them both on one command line, taskB will always run first. In my rollingUpgrade rule, I am able to correctly express my intent. If I’m doing a rolling upgrade, all the tasks must run and in the correct order. But I can still use any combination of the other rules on the command line (or use them to construct a different rule with different ordering requirements). This is useful if a deployment fails and I have to recover manually.