How to run the same job multiple times in parallel with Jenkins?

I understand you've looked into the Build Flow plugin, but I'm not sure why you've dismissed it. Perhaps you can point out the holes in my proposal.

Assuming you have enough executors in your system to run jobs in parallel, I think that the Build Flow plugin and Build Flow Test Aggregator plugin can do what you want.

• The Build Flow plugin supports running jobs in parallel. I don't see any reason why Build Flow could not schedule your "child" job to run in parallel with different parameters.

• The Build Flow Test Aggregator grabs test results from the scheduled builds of a Build Flow job, so your "child" job will need to publish its own test results.

• You will need to configure your "child" job so that it can run in parallel by checking the "Execute concurrent builds if necessary" in the job configuration.

• Whatever set of slaves provide the connection to the embedded devices will need enough executors to run your jobs in parallel.

---

Update: with the simple Build Flow definition:

parallel (
  { build("dbacher flow child", VALUE: 1) },
  { build("dbacher flow child", VALUE: 2) },
  { build("dbacher flow child", VALUE: 3) },
  { build("dbacher flow child", VALUE: 4) }
)

I get the output:

parallel {
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Build dbacher flow child #5 started
    Build dbacher flow child #6 started
    Build dbacher flow child #7 started
    Build dbacher flow child #8 started
    dbacher flow child #6 completed 
    dbacher flow child #7 completed 
    dbacher flow child #5 completed 
    dbacher flow child #8 completed 
}

The job history shows that all four jobs are scheduled within seconds of each other. But the job build step contains an artificial delay (sleep) that would prevent any single build from completing that quickly.

---

Update 2: Here is an example of generating the list of parallel tasks dynamically from another data structure:

// create a closure for the deploy job for each server 
def paramValues = (1..4)
def testJobs = [] 
for (param in paramValues) { 
  def jobParams = [VALUE: param] 
  def testJob = { 
    // call build 
    build(jobParams, "dbacher flow child") 
  } 
  println jobParams
  testJobs.add(testJob) 
} 

parallel(testJobs)

The list passed to parallel is a list of closures that call the build with unique parameters. I had to make sure to define the job parameters outside of the closure function to ensure the jobs would be scheduled separately.

I cribbed the syntax from another answer and this thread on the Jenkins mailing list.

Please make sure that the number of executors in the Manage Jenkins -> Manage Nodes settings is more than the number of individual jobs in MultiJob project. By default I guess it is 2. Hence we need to increase it.