Working with Jobs

Jobs are the executable units of your pipelines that can execute any DevOps activity. A simple way to think of it is, if something can execute in the shell of your laptop, it can execute as a Job.

Jobs take Inputs in the form of Resources, execute tasks that perform the operations necessary and then produce a result i.e. Output(s). Now these Outputs can become Inputs to other jobs and so on forming a dependency-based, event-driven DevOps Assembly Line.

Jobs are defined in a yml-based configuration file shippable.jobs.yml that is committed to source control in your Sync repository.

Adding Jobs

Jobs are defined in a configuration file shippable.jobs.yml and this file is added to the root of a source control repository. All user permissions that users have on the repo is carried over to the objects defined in the YML. For example, if user 1 has read access he/she will only have read access to Jobs defined in the repo.

Once the Jobs are defined and added to the repo, you will have to connect it to SPOG by creating a syncRepo using the UI. Detailed step by step instructions are here

After adding a syncRepo, our DevOps Assembly Lines are watching for changes (Job adds, edits or deleted) through source control webhooks. YML changes are automatically synced and they are reflected in the SPOG immediately

Migrating a Job from one YML to another

There are some situations where you might need to reorganize where your Jobs are defined. If you delete and recreate, you will lose all the historical versions. If history is important, migration might be a better alternative

Here are the steps to migrate -

Here are the steps to migrate -

  1. Goto the SPOG page and pause the rSync job for the source repository from where you want to migrate resources, jobs or triggers. Steps for pausing the rSync jobs can be found here.

  2. Add the jobs you want to migrate to the destination repository's yml files.

  3. Run the rSync job of the destination repository. Once the rSync job completes, migration will be complete. Your SPOG will however not change as no dependency has changed. Therefore, to verify this you can check the logs of the rSync job. See the version list of your resource from the SPOG view

  4. From the source repository, delete all the migrated jobs. This will automatically trigger the rSync job and it should succeed. See the version list of your resource from the SPOG view

  5. Now, your jobs are migrated to the destination repository. You can add, remove or update them there.

NOTE: While migrating a job you should copy it exactly as it is in the source repository. You can add or remove dependencies only once the job is successfully migrated to destination repository.

Pausing jobs

You can pause any jobs in your pipeline by right-clicking on the job, and clicking Pause Jobs. Paused jobs are never triggered automatically, irrespective of yml configuration. You can unpause a paused job to resume automatic triggers.

Pause job

Turning off automatic job execution

You can insert a manual approval gate by preventing a job from being triggered automatically. The config for this is shown below:

jobs:
  - name: Job-3
    type: release
    steps:
      - IN: resource-2
        switch: off
      - IN: Job-1
        switch: off

As shown above, the switch: off tag can be defined for IN resources or jobs in order to turn off automatic triggering of a job when the inputs change. Note that these switches are specific to the input and should be set accordingly for each input.

Pinning specific resource versions

By default, Shippable uses information from the most recent or latest version of an IN input when running a job. However, you might want to 'pin' a specific version of an input for some reason. Input versions may be pinned either though the yml configuration or in the UI.

Pinning resource versions in shippable.jobs.yml

You can pin a specific input version with the yml below:

jobs:
  - name: job_name
    type: job_type
    steps:
      - IN: resource-1
        versionName: "user friendly version e.g tag or commitSha"
      - IN: resource-2
        versionNumber: "shippable's internal version number"

You can use versionName to pin gitRepo and image resources. The versionName contains:

  • gitRepo: commit SHA
  • image: tag

You can use versionNumber, Shippable's internal incremental numbering system, to pin the following resources:

  • dockerOptions
  • params
  • replicas

Pinning versions in the UI

To pin a version of an input resource in the UI, first right-click on the job and click Configure Job. This will open a configuration page where you can select a version to pin for any of the inputs. Versions may be unpinned on the same page by selecting Latest version.

Deleting Jobs

Deleting Job is a 2 step process. Deleting from the YML causes it to be soft deleted and then you will have to manually delete it from SPOG view of the UI. The 2 step process is an insurance policy to prevent accidental deletes. Deleting a Job means all historical versions are deleted permanently and this can really mess up your DevOps Assembly Lines as it is a connected interdependent workflow.

Here are the steps to delete a Job

  1. Delete the Job definition from the YML and commit the change to the repo. Automatic sync process will execute and mark the removed Job as soft deleted. All version data is still available and no data is lost at this stage. If any Jobs are still using this deleted Job and they are still active, those Jobs will be marked inconsistent and will not be triggered until fixed
  2. Now log into your SPOG, make sure that you have not hidden deleted Job from your view. All the soft deleted Jobs will appear on the bottom of your SPOG. Right click on each of them and delete. If these Jobs are still being used, then delete option is not presented until you make sure there are no references to the soft deleted Job. Once you delete from the SPOG, all data is permanently destroyed

Further Reading