Break out 'build steps' into individual job 'plugins'.

One concern with the above structure is that it doesn't accommodate multiple invocations of the same job type, as this would lead to conflicts on the properties array keys. Creating seperate 'setup', 'build', and 'execute' plugins doesn't make much sense, if all three of these perform the same action.

An alternative approach:

array(
  'job_tasks' => array(
    '0' => array(
      // Setup Commands
      'plugin' => 'execute',
      'properties' => array(...),
    ),
    '1' => array(
      // Git Checkout
      'plugin' => 'vcs',
      'properties' => array(...),
    ),
    '2' => array(
      // Apply Patch
      'plugin' => 'patch',
      'properties' => array(...),
    ),
    '3' => array(
      // Post-build steps
      'plugin' => 'execute',
      'properties' => array(...),
    ),
  ),
)

Disadvantage to #1 is that it requires duplication of properties for tasks which use the same property definitions ... another option is to keep the properties seperate from the job stack, as per #1 ... but allow a mechanism for overriding the array key associated with a particular item in the job stack.

array(
  'job_tasks' => array(
    '0' => array(
      'plugin' => 'execute',
      'properties' => 'execute1',
    ),
    '1' => array(
      'plugin' => 'vcs',
    ),
    '2' => array(
      'plugin' => 'patch',
    ),
    '3' => array(
      'plugin' => 'execute',
      'properties' => 'execute2',
    ),
  ),
  'execute1' => array(...),
  'execute2' => array(...),
  'vcs' => array(...),
  'patch' => array(...),
   ...

Admittedly, the extra array() in the job tasks is a bit annoying ... but this could be made optional; cleaning up the array (but adding complexity to the processing code).

array(
  'job_tasks' => array(
    '0' => array(
      'plugin' => 'execute',
      'properties' => 'execute1',
    ),
    '1' => 'vcs',
    '2' => 'patch',
    '3' => array(
      'plugin' => 'execute',
      'properties' => 'execute2',
    ),
  ),
  'execute1' => array(...),
  'execute2' => array(...),
  'vcs' => array(...),
  'patch' => array(...),
   ...

The property name 'task' makes more sense than 'job_task', as 'job' has a certain context in the current build.

Next challenge is how to handle those build steps or plugins which depend on other properties as well. For example, the 'patch' processing also contains a dependency on the 'vcs' property.

One proposed approach is related to #1649986: Allow each plugin to define the complete set of 'allowed' properties it accepts. Adding a capability for each plugin to define the complete set of properties that it depends on would allow us to simply iterate over that list, look for any keys in the properties array which match, and forward those along to the plugin.

This could conflict with the second structure defined in #2 above, as you would not be able to match a 'user-named' property against the property named in the plugin definition. This combination of 'multiple plugin invocations' combined with a patch dependency could be considered a bit of an edge case ... but would still need to be accomodated. An option here is to map the 'plugin' defined property name to the 'user' defined property name in the task array.

array(
  'job_tasks' => array(
    '0' => 'setup',
    '1' => array(
      'plugin' => 'vcs',
      'properties' => 'vcs1',
    ),
    '2' => array(
      'plugin' => 'vcs',
      'properties' => 'vcs2',
    ),
    '3' => array(
      'plugin' => 'patch',
      'vcs' => 'vcs2',
    ),
    '4' => 'build',
  ),
  'setup' => array(...),
  'vcs1' => array(...),
  'vcs2' => array(...),
  'patch' => array(...),
  'build' => array(...),
   ...

First cut on the worker side.

Updated issue summary.

Second iteration. Includes a one-line patch to default properties on the conduit side.

Missing required argument 'status'.

Minor fix.

This patch isn't compatible with the existing conduit_drupal plugins, as they would need to add their own default 'tasks' entry into the properties array.

Updated issue summary.