The DevOps toolchain is fragmented and most teams use multiple tools to automate specific activities like CI, release orchestration, deployments, infrastructure management, etc. This creates "Islands of Automation" since these tools cannot natively talk to each other to exchange relevant information. These boundaries between teams and tools are often handled with manual processes like tickets or Slack channels. Shippable solves this problem by providing an easy way to connect your islands of automation into continuous, streamlined, stateful workflows.
State(ful) workflows are designed to remember information generated by every job and resource and make it available to dependent jobs or successive executions of the same job. This is a crucial component in order to achieve end-to-end Continuous Delivery.
Some example use-cases are the following:
- A CI job creates information about the commitSha and image/file version that was built, which is then consumed by another job to deploy the correct version into the test environment.
- A job creates a VPC for the Staging environment. It stores information like VPC info, subnets, security groups, etc. which is required by another job that deploys to the Staging environment.
- You have a provisioning job that uses Terraform to create the Test environment. At a later stage in your Assembly Line, you have a deprovisioning job that destroys the test environment. Both these jobs need to read and update the Terraform state file so they are aware of the current state of the Test environment.
You can use state as you need, depending on your workflow. At the end of the day, it is just a way for jobs in your Assembly Line to exchange information.
Types of state
The Assembly Lines platform supports three types of state: job-based state, resource-based state, and central state. Job-based state allows you to share key-value pairs between connected jobs and subsequent runs of the same job. Resource-based state allows you to share key-value pairs or files across any job in your Assembly Line. Central state allows you to share files or key-value pairs and is very useful in situations where multiple jobs in your Assembly Line need to consume as well as update the data.
You can store key-value pairs or files up to 1Mb as state for every immutable version, i.e. run, of a job. This state is available to any connected downstream jobs, or to subsequent runs of the same job. This makes it easy to transfer state-ful information from one job to another or across runs.
To learn how to store key-value state or files in job state, read our tutorial on Read/write from job-based-state
Key-values can be stored in a params resource or as a property in the immutable version of any resource. Every job that has the resource or job as an
IN can access the key-value information in its scripts.
For an example, check out our tutorials:
The scenarios above are simple since one job creates/updates state and another job or subsequent runs of the same job consume the state. However, you will sometimes need to store files or key-value pairs that are changed and maintained by several jobs in your Assembly Line.
For example, Terraform creates a statefile when
terraform apply is run, and subsequent jobs in your Assembly Line might want to change this statefile. These updates now need to flow back to your earlier job so it is aware of the latest statefile. Using the methods above would create a circular reference which is not allowed for jobs or resources in an Assembly Line.
state resource is key for these scenarios since it is the only resource that can create a circular loop without creating problems in your Assembly Line. This is because the
state resource never triggers the job automatically even when it changes. It just maintains central state for the jobs that need them and make it available when required. You can store key-value pairs or files up to 1MB in the