Pfthb

I'm trying to dynamically create jobs in a Jenkinsfile using the following loop. The jobs are created correctly, and the task names show up in Jenkins with the correct name (e.g. ubuntu:bionic).

However, the within each task the sh commands seem to not have access to images as $images[i] is being evaluated to null (e.g. sh ci/script.sh null). So this doesn't seem to be the same as other single-vs-double quote issues with string interpolation.

def images = ["ubuntu:bionic", "ubuntu:xenial"]
def tasks = [:]

for (int i = 0; i < images.size(); i++) 
 tasks["$images[i]"] = 
 node 
 lock("build") 
 stage('checkout') 
 checkout scm
 
 stage('test') 
 sh "ci/script.sh $images[i]"
 
 
 
 


stage("matrix") 
 parallel tasks

How can I build these dynamic commands correctly?

A closure you create in the loop and assign to tasks["$images[i]"] is evaluated lazily and it seems like it processes images.getAt(i) with the current i value, which in this case is equal to 2 in both cases. Take a look at following example with some additional printing of current i state (I've skipped scm checkout int this short example):

def images = ["ubuntu:bionic", "ubuntu:xenial"]
def tasks = [:]

for (int i = 0; i < images.size(); i++) 
 println "Using i = $i" // <- first print
 tasks["$images[i]"] = 
 node 
 lock("build") 
 stage('checkout') 
 echo "ok"
 
 stage('test') 
 println "Print i inside stage = $i" // <- second print
 echo "Echo i inside stage = $i" // <- third print
 sh "ci/script.sh $images[i]".toString()
 
 
 
 


stage("matrix") 
 parallel tasks

When we run it we will see something like this in the console:

[Pipeline] echo
Using i = 0
[Pipeline] echo
Using i = 1
[Pipeline] stage
[Pipeline] (matrix)
[Pipeline] parallel
[Pipeline] [ubuntu:bionic] (Branch: ubuntu:bionic)
[Pipeline] [ubuntu:xenial] (Branch: ubuntu:xenial)
[Pipeline] [ubuntu:bionic] node
[ubuntu:bionic] Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] [ubuntu:xenial] node
[ubuntu:xenial] Running on Jenkins in /var/jenkins_home/workspace/test-pipeline@2
[Pipeline] [ubuntu:bionic] 
[Pipeline] [ubuntu:xenial] 
[Pipeline] [ubuntu:bionic] lock
[ubuntu:bionic] Trying to acquire lock on [build]
[ubuntu:bionic] Lock acquired on [build]
[Pipeline] [ubuntu:bionic] 
[Pipeline] [ubuntu:xenial] lock
[ubuntu:xenial] Trying to acquire lock on [build]
[ubuntu:xenial] Found 0 available resource(s). Waiting for correct amount: 1.
[ubuntu:xenial] [build] is locked, waiting...
[Pipeline] [ubuntu:bionic] stage
[Pipeline] [ubuntu:bionic] (checkout)
[Pipeline] [ubuntu:bionic] echo
[ubuntu:bionic] ok
[Pipeline] [ubuntu:bionic] 
[Pipeline] [ubuntu:bionic] // stage
[Pipeline] [ubuntu:bionic] stage
[Pipeline] [ubuntu:bionic] (test)
[Pipeline] [ubuntu:bionic] echo
[ubuntu:bionic] Print i inside stage = 2
[Pipeline] [ubuntu:bionic] echo
[ubuntu:bionic] Echo i inside stage = 2
[Pipeline] [ubuntu:bionic] sh
[ubuntu:bionic] [test-pipeline] Running shell script
[ubuntu:bionic] + ci/script.sh null
[ubuntu:bionic] /var/jenkins_home/workspace/test-pipeline@tmp/durable-998289d1/script.sh: 2: /var/jenkins_home/workspace/test-pipeline@tmp/durable-998289d1/script.sh: ci/script.sh: not found
[Pipeline] [ubuntu:bionic] 
[Pipeline] [ubuntu:bionic] // stage
[ubuntu:xenial] Lock acquired on [build]
[Pipeline] [ubuntu:bionic] 
[ubuntu:bionic] Lock released on resource [build]
[Pipeline] [ubuntu:xenial] 
[Pipeline] [ubuntu:bionic] // lock
[Pipeline] [ubuntu:bionic] 
[Pipeline] [ubuntu:xenial] stage
[Pipeline] [ubuntu:xenial] (checkout)
[Pipeline] [ubuntu:bionic] // node
[Pipeline] [ubuntu:bionic] 
[ubuntu:bionic] Failed in branch ubuntu:bionic
[Pipeline] [ubuntu:xenial] echo
[ubuntu:xenial] ok
[Pipeline] [ubuntu:xenial] 
[Pipeline] [ubuntu:xenial] // stage
[Pipeline] [ubuntu:xenial] stage
[Pipeline] [ubuntu:xenial] (test)
[Pipeline] [ubuntu:xenial] echo
[ubuntu:xenial] Print i inside stage = 2
[Pipeline] [ubuntu:xenial] echo
[ubuntu:xenial] Echo i inside stage = 2
[Pipeline] [ubuntu:xenial] sh
[ubuntu:xenial] [test-pipeline@2] Running shell script
[ubuntu:xenial] + ci/script.sh null
[ubuntu:xenial] /var/jenkins_home/workspace/test-pipeline@2@tmp/durable-b1807fa2/script.sh: 2: /var/jenkins_home/workspace/test-pipeline@2@tmp/durable-b1807fa2/script.sh: ci/script.sh: not found
[Pipeline] [ubuntu:xenial] 
[Pipeline] [ubuntu:xenial] // stage
[Pipeline] [ubuntu:xenial] 
[ubuntu:xenial] Lock released on resource [build]
[Pipeline] [ubuntu:xenial] // lock
[Pipeline] [ubuntu:xenial] 
[Pipeline] [ubuntu:xenial] // node
[Pipeline] [ubuntu:xenial] 
[ubuntu:xenial] Failed in branch ubuntu:xenial
[Pipeline] // parallel
[Pipeline] 
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE

I have used println inside the stage on purpose, because this is not a Jenkins pipeline step, but simple Groovy method. As you can see it gets evaluated when the parallel execution happens in matrix stage. Each Groovy closure is associated with bindings - its local state of variables. It looks like it contains images and i bindings and it tracks changes of the state of i variable. That is why it tries to access images[2] when evaluating sh step.

Solution

There is a simple solution to this problem. You can replace for-loop with for-each. Consider following example:

def images = ["ubuntu:bionic", "ubuntu:xenial"]
def tasks = [:]

images.each image ->
 tasks["$image"] = 
 node 
 lock("build") 
 stage('checkout') 
 checkout scm
 
 stage('test') 
 sh "ci/script.sh $image"
 
 
 
 


stage("matrix") 
 parallel tasks

Console output:

[ubuntu:bionic] [test-pipeline] Running shell script
[ubuntu:bionic] + ci/script.sh ubuntu:bionic
[ubuntu:xenial] [test-pipeline@2] Running shell script
[ubuntu:xenial] + ci/script.sh ubuntu:xenial

You can find explanation of global scope of i variable in Pipeline - Parallel execution of tasks article on CloudBees:

Note: Variables define outside the for block are not local, but global to the script. Testing the option 2, you will notice that variable i prints always value 4, whereas index increases from 0 to 3 and branch from 1 to 4.