Jobs

Jobs are the executable units of your pipelines that can execute any DevOps activity. A simple way to think of it is, if something can execute in the shell of your laptop, it can execute as a Job.

Jobs take Inputs in the form of Resources, execute tasks that perform the operations necessary and then produce a result i.e. Output(s). Now these Outputs can become Inputs to other jobs and so on forming a dependency-based, event-driven DevOps Assembly Line.

Typical examples of jobs are:

• Execute Continuous Integration (CI) on a repository
• Build, Test and Push Docker images to registries like Docker Hub. Quay, JFrog etc.
• Build deployable machine images with tools like Packer
• Provisioning infrastructure using various automation tools like Ansible, Terraform, Chef, Puppet, etc.
• Deploy applications to Container Orchestration Services like Kubernetes, Amazon ECS, Google Container Engine, Azure Container Service,
• Deploy an application version to a VM cluster
• Execute a functional/integration test suite
• Create immutable service definitions so that applications can roll-forward and roll-back
• Deploy applications with Continuous Deployment(CD) tools like Capistrano, Distelli, Ansible etc.
• Implementing deployment strategies like Blue/Green, Upgrades and Replace
• Perform timely backups of DB, key-value stores etc.
• Create or update an application release version

YML Definition

Jobs are defined in the config file shippable.resources.yml with the syntax below. Please note that this is the complete syntax and an actual job definition is much smaller than shown here.

jobs:
  - name:                   <string>
    type:                   <job type name>
     on_start:
      - NOTIFY:             <notification resource name>
    steps:
      - IN:                 <resource>
        switch:             off
      - IN:                 <job>
      - IN:                 <resource>
        versionName:        <name of the version you want to pin>
      - IN:                 <resource>
        versionNumber:      <number of the version you want to pin>
      - IN:                 <params/dockerOption/replicas resource>
        applyTo:
          -                 <image/manifest/release>
          -                 <image/manifest/release>
      - IN:                 <loadBalancer resource>
        applyTo:
          - manifest:       <manifest>
            image:          <image>              
            port:           <number>
      - IN:                 <gitRepoResource with buildOnPullRequest: true>
        showBuildStatus:    true
      - IN:                 <manifest/release>
        force:              true
      - TASK:
        - script:           <any shell command>
        - script:           <any shell command>
      - OUT:                <resource>
      - OUT:                <resource>
        replicate:          <IN resource>
      - OUT:                <resource>
        overwrite:          true
     on_success:
      - script:             echo "SUCCESS"
     on_failure:
      - script:             echo "FAILED"
      - NOTIFY:             <notification resource name>
     on_cancel:
      - script:             echo "CANCEL"
     always:
      - script:             pwd

For more information, read Job Anatomy

When does a Job execute?

Jobs can be triggered in multiple ways, including both automated and manual triggers. For example, consider the configuration below for my-deploy:

In this configuration, Job-4 will be triggered in one of 4 ways:

• A previous job in the Assembly Line changes an IN resource, i.e. Job-4 will be triggered every time Job-3 updates resource-2.
• An IN job successfully completed execution, i.e. Job-4 will be triggered every time Job-2 succeeds. A couple of exceptions to this rule are:
  • If any Jobs defined as INs are currently processing, have queued builds or are in a failed state. So in the example above, if Job-1 is in failed state or is queued, Job-4 will not trigger even when Job-2 completes successfully.
  • If the Job is in inconsistent state due to dependency failures
• You commit to a trigger that is an IN for the job, i.e. trigger-1 in this example
• Manual trigger: You right click on Job-4 in the SPOG UI and click on Run or Select Run for Job-4 in the Jobs list in the Grid view.

Job-1 will not trigger Job-4 automatically since the dotted line between them indicates that automatic trigger has been switched off.

Please note that changing some resources like cluster, replicas, or dockerOptions manually through a yml commit will not automatically trigger a job. This behavior is meant to prevent unexpected pipeline behavior, since a single commit can contains changes to several resources and cause several trigger points in the pipeline. If you want your job to be triggered when resources are manually edited in the yml, you can add a trigger input for the job and include a change to the trigger resource in the commit every time you want to automatically run your job.

What happens when a Job executes?

When triggered, a Job does the following:

• Get the latest versions of IN resources or latest state information from an IN job. If an input resource version is pinned, then use that version instead of latest.
• Execute all commands to perform the activity. The set of commands depends on job type.
• Update all OUT resources

Every time a job executes, a new immutable version is created. This makes it easy to 'replay' older job versions, though you should do this only after verifying that the old input values will not create problems in your Assembly Line.

Supported job types

Shippable supports the following jobs types:

If you need a job that is not listed above, send us an email at support@shippable.com

Further Reading

• Anatomy of a Job yml
• Working with jobs
• Resources
• Quick Start to CI
• Quick Start to CD