Understanding OpenShift Pipelines
Understanding OpenShift Pipelines
Red Hat OpenShift Pipelines is a cloud-native, continuous integration and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces a number of standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.
Key features
Red Hat OpenShift Pipelines is a serverless CI/CD system that runs pipelines with all the required dependencies in isolated containers.
Red Hat OpenShift Pipelines are designed for decentralized teams that work on microservice-based architecture.
Red Hat OpenShift Pipelines use standard CI/CD pipeline definitions that are easy to extend and integrate with the existing Kubernetes tools, enabling you to scale on-demand.
You can use Red Hat OpenShift Pipelines to build images with Kubernetes tools such as Source-to-Image (S2I), Buildah, Buildpacks, and Kaniko that are portable across any Kubernetes platform.
You can use the OpenShift Container Platform Developer console to create Tekton resources, view logs of pipeline runs, and manage pipelines in your OpenShift Container Platform namespaces.
OpenShift Pipelines Concepts
This guide provides a detailed view of the various pipeline concepts.
Tasks
Task resources are the building blocks of a pipeline and consist of sequentially executed steps. It is essentially a function of inputs and outputs. A task can run individually or as a part of the pipeline. Tasks are reusable and can be used in multiple pipelines.
Steps are a series of commands that are sequentially executed by the task and achieve a specific goal, such as building an image. Every task runs as a pod, and each step runs as a container within that pod. Because steps run within the same pod, they can access the same volumes for caching files, config maps, and secrets.
The following example shows the apply-manifests task.
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: apply-manifests
spec:
workspaces:
- name: source
params:
- name: manifest_dir
description: The directory in source that contains yaml manifests
type: string
default: "k8s"
steps:
- name: apply
image: image-registry.openshift-image-registry.svc:5000/openshift/cli:latest
workingDir: /workspace/source
command: ["/bin/bash", "-c"]
args:
- |-
echo Applying manifests in $(params.manifest_dir) directory
oc apply -f $(params.manifest_dir)
echo -----------------------------------The task API version, v1. | |
The type of Kubernetes object, Task. | |
| The unique name of this task. | |
| The list of parameters and steps in the task and the workspace used by the task. |
This task starts the pod and runs a container inside that pod using the specified image to run the specified commands.
Starting with OpenShift Pipelines 1.6, the following defaults from the step YAML file are removed:
Instead, the container for the step defines the As a temporary measure, to maintain backward compatibility with the older OpenShift Pipelines versions, you can set the following fields in the |
When expression
When expressions guard task execution by setting criteria for the execution of tasks within a pipeline. They contain a list of components that allows a task to run only when certain criteria are met. When expressions are also supported in the final set of tasks that are specified using the finally field in the pipeline YAML file.
The key components of a when expression are as follows:
input: Specifies static inputs or variables such as a parameter, task result, and execution status. You must enter a valid input. If you do not enter a valid input, its value defaults to an empty string.operator: Specifies the relationship of an input to a set ofvalues. Enterinornotinas your operator values.values: Specifies an array of string values. Enter a non-empty array of static values or variables such as parameters, results, and a bound state of a workspace.
The declared when expressions are evaluated before the task is run. If the value of a when expression is True, the task is run. If the value of a when expression is False, the task is skipped.
You can use the when expressions in various use cases. For example, whether:
The result of a previous task is as expected.
A file in a Git repository has changed in the previous commits.
An image exists in the registry.
An optional workspace is available.
The following example shows the when expressions for a pipeline run. The pipeline run will execute the create-file task only if the following criteria are met: the path parameter is README.md, and the echo-file-exists task executed only if the exists result from the check-file task is yes.
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: guarded-pr-
spec:
taskRunTemplate:
serviceAccountName: pipeline
pipelineSpec:
params:
- name: path
type: string
description: The path of the file to be created
workspaces:
- name: source
description: |
This workspace is shared among all the pipeline tasks to read/write common resources
tasks:
- name: create-file
when:
- input: "$(params.path)"
operator: in
values: ["README.md"]
workspaces:
- name: source
workspace: source
taskSpec:
workspaces:
- name: source
description: The workspace to create the readme file in
steps:
- name: write-new-stuff
image: ubuntu
script: 'touch $(workspaces.source.path)/README.md'
- name: check-file
params:
- name: path
value: "$(params.path)"
workspaces:
- name: source
workspace: source
runAfter:
- create-file
taskSpec:
params:
- name: path
workspaces:
- name: source
description: The workspace to check for the file
results:
- name: exists
description: indicates whether the file exists or is missing
steps:
- name: check-file
image: alpine
script: |
if test -f $(workspaces.source.path)/$(params.path); then
printf yes | tee /tekton/results/exists
else
printf no | tee /tekton/results/exists
fi
- name: echo-file-exists
when:
- input: "$(tasks.check-file.results.exists)"
operator: in
values: ["yes"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: 'echo file exists'
...
- name: task-should-be-skipped-1
when:
- input: "$(params.path)"
operator: notin
values: ["README.md"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: exit 1
...
finally:
- name: finally-task-should-be-executed
when:
- input: "$(tasks.echo-file-exists.status)"
operator: in
values: ["Succeeded"]
- input: "$(tasks.status)"
operator: in
values: ["Succeeded"]
- input: "$(tasks.check-file.results.exists)"
operator: in
values: ["yes"]
- input: "$(params.path)"
operator: in
values: ["README.md"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: 'echo finally done'
params:
- name: path
value: README.md
workspaces:
- name: source
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 16MiSpecifies the type of Kubernetes object. In this example, PipelineRun. | |
Task create-file used in the pipeline. | |
when expression that specifies to execute the echo-file-exists task only if the exists result from the check-file task is yes. | |
when expression that specifies to skip the task-should-be-skipped-1 task only if the path parameter is README.md. | |
when expression that specifies to execute the finally-task-should-be-executed task only if the execution status of the echo-file-exists task and the task status is Succeeded, the exists result from the check-file task is yes, and the path parameter is README.md. |
The Pipeline Run details page of the OpenShift Container Platform web console shows the status of the tasks and when expressions as follows:
All the criteria are met: Tasks and the when expression symbol, which is represented by a diamond shape are green.
Any one of the criteria are not met: Task is skipped. Skipped tasks and the when expression symbol are grey.
None of the criteria are met: Task is skipped. Skipped tasks and the when expression symbol are grey.
Task run fails: Failed tasks and the when expression symbol are red.
Finally tasks
The finally tasks are the final set of tasks specified using the finally field in the pipeline YAML file. A finally task always executes the tasks within the pipeline, irrespective of whether the pipeline runs are executed successfully. The finally tasks are executed in parallel after all the pipeline tasks are run, before the corresponding pipeline exits.
You can configure a finally task to consume the results of any task within the same pipeline. This approach does not change the order in which this final task is run. It is executed in parallel with other final tasks after all the non-final tasks are executed.
The following example shows a code snippet of the clone-cleanup-workspace pipeline. This code clones the repository into a shared workspace and cleans up the workspace. After executing the pipeline tasks, the cleanup task specified in the finally section of the pipeline YAML file cleans up the workspace.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: clone-cleanup-workspace
spec:
workspaces:
- name: git-source
tasks:
- name: clone-app-repo
taskRef:
name: git-clone-from-catalog
params:
- name: url
value: https://github.com/tektoncd/community.git
- name: subdirectory
value: application
workspaces:
- name: output
workspace: git-source
finally:
- name: cleanup
taskRef:
name: cleanup-workspace
workspaces:
- name: source
workspace: git-source
- name: check-git-commit
params:
- name: commit
value: $(tasks.clone-app-repo.results.commit)
taskSpec:
params:
- name: commit
steps:
- name: check-commit-initialized
image: alpine
script: |
if [[ ! $(params.commit) ]]; then
exit 1
fi| Unique name of the pipeline. | |
| The shared workspace where the git repository is cloned. | |
| The task to clone the application repository to the shared workspace. | |
| The task to clean-up the shared workspace. | |
| A reference to the task that is to be executed in the task run. | |
| A shared storage volume that a task in a pipeline needs at runtime to receive input or provide output. | |
| A list of parameters required for a task. If a parameter does not have an implicit default value, you must explicitly set its value. | |
| Embedded task definition. |
TaskRun
A TaskRun instantiates a task for execution with specific inputs, outputs, and execution parameters on a cluster. It can be invoked on its own or as part of a pipeline run for each task in a pipeline.
A task consists of one or more steps that execute container images, and each container image performs a specific piece of build work. A task run executes the steps in a task in the specified order, until all steps execute successfully or a failure occurs. A TaskRun is automatically created by a PipelineRun for each task in a pipeline.
The following example shows a task run that runs the apply-manifests task with the relevant input parameters:
apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
name: apply-manifests-taskrun
spec:
taskRunTemplate:
serviceAccountName: pipeline
taskRef:
kind: Task
name: apply-manifests
workspaces:
- name: source
persistentVolumeClaim:
claimName: source-pvcThe task run API version v1. | |
Specifies the type of Kubernetes object. In this example, TaskRun. | |
| Unique name to identify this task run. | |
| Definition of the task run. For this task run, the task and the required workspace are specified. | |
Name of the task reference used for this task run. This task run executes the apply-manifests task. | |
| Workspace used by the task run. |
Pipelines
A Pipeline is a collection of Task resources arranged in a specific order of execution. They are executed to construct complex workflows that automate the build, deployment and delivery of applications. You can define a CI/CD workflow for your application using pipelines containing one or more tasks.
A Pipeline resource definition consists of a number of fields or attributes, which together enable the pipeline to accomplish a specific goal. Each Pipeline resource definition must contain at least one Task resource, which ingests specific inputs and produces specific outputs. The pipeline definition can also optionally include Conditions, Workspaces, Parameters, or Resources depending on the application requirements.
The following example shows the build-and-deploy pipeline, which builds an application image from a Git repository using the buildah ClusterTask resource:
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: build-and-deploy
spec:
workspaces:
- name: shared-workspace
params:
- name: deployment-name
type: string
description: name of the deployment to be patched
- name: git-url
type: string
description: url of the git repo for the code of deployment
- name: git-revision
type: string
description: revision to be used from repo of the code for deployment
default: "pipelines-1.14"
- name: IMAGE
type: string
description: image to be built from the code
tasks:
- name: fetch-repository
taskRef:
name: git-clone
kind: ClusterTask
workspaces:
- name: output
workspace: shared-workspace
params:
- name: url
value: $(params.git-url)
- name: subdirectory
value: ""
- name: deleteExisting
value: "true"
- name: revision
value: $(params.git-revision)
- name: build-image
taskRef:
name: buildah
kind: ClusterTask
params:
- name: TLSVERIFY
value: "false"
- name: IMAGE
value: $(params.IMAGE)
workspaces:
- name: source
workspace: shared-workspace
runAfter:
- fetch-repository
- name: apply-manifests
taskRef:
name: apply-manifests
workspaces:
- name: source
workspace: shared-workspace
runAfter:
- build-image
- name: update-deployment
taskRef:
name: update-deployment
workspaces:
- name: source
workspace: shared-workspace
params:
- name: deployment
value: $(params.deployment-name)
- name: IMAGE
value: $(params.IMAGE)
runAfter:
- apply-manifestsPipeline API version v1. | |
Specifies the type of Kubernetes object. In this example, Pipeline. | |
| Unique name of this pipeline. | |
| Specifies the definition and structure of the pipeline. | |
| Workspaces used across all the tasks in the pipeline. | |
| Parameters used across all the tasks in the pipeline. | |
| Specifies the list of tasks used in the pipeline. | |
Task build-image, which uses the buildah ClusterTask to build application images from a given Git repository. | |
Task apply-manifests, which uses a user-defined task with the same name. | |
Specifies the sequence in which tasks are run in a pipeline. In this example, the apply-manifests task is run only after the build-image task is completed. |
The Red Hat OpenShift Pipelines Operator installs the Buildah cluster task and creates the |
PipelineRun
A PipelineRun is a type of resource that binds a pipeline, workspaces, credentials, and a set of parameter values specific to a scenario to run the CI/CD workflow.
A pipeline run is the running instance of a pipeline. It instantiates a pipeline for execution with specific inputs, outputs, and execution parameters on a cluster. It also creates a task run for each task in the pipeline run.
The pipeline runs the tasks sequentially until they are complete or a task fails. The status field tracks and the progress of each task run and stores it for monitoring and auditing purposes.
The following example runs the build-and-deploy pipeline with relevant resources and parameters:
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
name: build-deploy-api-pipelinerun
spec:
pipelineRef:
name: build-and-deploy
params:
- name: deployment-name
value: vote-api
- name: git-url
value: https://github.com/openshift-pipelines/vote-api.git
- name: IMAGE
value: image-registry.openshift-image-registry.svc:5000/pipelines-tutorial/vote-api
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500MiPipeline run API version v1. | |
The type of Kubernetes object. In this example, PipelineRun. | |
| Unique name to identify this pipeline run. | |
Name of the pipeline to be run. In this example, build-and-deploy. | |
| The list of parameters required to run the pipeline. | |
| Workspace used by the pipeline run. |
Workspaces
It is recommended that you use workspaces instead of the |
Workspaces declare shared storage volumes that a task in a pipeline needs at runtime to receive input or provide output. Instead of specifying the actual location of the volumes, workspaces enable you to declare the filesystem or parts of the filesystem that would be required at runtime. A task or pipeline declares the workspace and you must provide the specific location details of the volume. It is then mounted into that workspace in a task run or a pipeline run. This separation of volume declaration from runtime storage volumes makes the tasks reusable, flexible, and independent of the user environment.
With workspaces, you can:
Store task inputs and outputs
Share data among tasks
Use it as a mount point for credentials held in secrets
Use it as a mount point for configurations held in config maps
Use it as a mount point for common tools shared by an organization
Create a cache of build artifacts that speed up jobs
You can specify workspaces in the TaskRun or PipelineRun using:
A read-only config map or secret
An existing persistent volume claim shared with other tasks
A persistent volume claim from a provided volume claim template
An
emptyDirthat is discarded when the task run completes
The following example shows a code snippet of the build-and-deploy pipeline, which declares a shared-workspace workspace for the build-image and apply-manifests tasks as defined in the pipeline.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: build-and-deploy
spec:
workspaces:
- name: shared-workspace
params:
...
tasks:
- name: build-image
taskRef:
name: buildah
kind: ClusterTask
params:
- name: TLSVERIFY
value: "false"
- name: IMAGE
value: $(params.IMAGE)
workspaces:
- name: source
workspace: shared-workspace
runAfter:
- fetch-repository
- name: apply-manifests
taskRef:
name: apply-manifests
workspaces:
- name: source
workspace: shared-workspace
runAfter:
- build-image
...List of workspaces shared between the tasks defined in the pipeline. A pipeline can define as many workspaces as required. In this example, only one workspace named shared-workspace is declared. | |
Definition of tasks used in the pipeline. This snippet defines two tasks, build-image and apply-manifests, which share a common workspace. | |
List of workspaces used in the build-image task. A task definition can include as many workspaces as it requires. However, it is recommended that a task uses at most one writable workspace. | |
Name that uniquely identifies the workspace used in the task. This task uses one workspace named source. | |
Name of the pipeline workspace used by the task. Note that the workspace source in turn uses the pipeline workspace named shared-workspace. | |
List of workspaces used in the apply-manifests task. Note that this task shares the source workspace with the build-image task. |
Workspaces help tasks share data, and allow you to specify one or more volumes that each task in the pipeline requires during execution. You can create a persistent volume claim or provide a volume claim template that creates a persistent volume claim for you.
The following code snippet of the build-deploy-api-pipelinerun pipeline run uses a volume claim template to create a persistent volume claim for defining the storage volume for the shared-workspace workspace used in the build-and-deploy pipeline.
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
name: build-deploy-api-pipelinerun
spec:
pipelineRef:
name: build-and-deploy
params:
...
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi| Specifies the list of pipeline workspaces for which volume binding will be provided in the pipeline run. | |
| The name of the workspace in the pipeline for which the volume is being provided. | |
| Specifies a volume claim template that creates a persistent volume claim to define the storage volume for the workspace. |
Triggers
Use Triggers in conjunction with pipelines to create a full-fledged CI/CD system where Kubernetes resources define the entire CI/CD execution. Triggers capture the external events, such as a Git pull request, and process them to extract key pieces of information. Mapping this event data to a set of predefined parameters triggers a series of tasks that can then create and deploy Kubernetes resources and instantiate the pipeline.
For example, you define a CI/CD workflow using Red Hat OpenShift Pipelines for your application. The pipeline must start for any new changes to take effect in the application repository. Triggers automate this process by capturing and processing any change event and by triggering a pipeline run that deploys the new image with the latest changes.
Triggers consist of the following main resources that work together to form a reusable, decoupled, and self-sustaining CI/CD system:
The
TriggerBindingresource extracts the fields from an event payload and stores them as parameters.The following example shows a code snippet of the
TriggerBindingresource, which extracts the Git repository information from the received event payload:apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerBinding metadata: name: vote-app spec: params: - name: git-repo-url value: $(body.repository.url) - name: git-repo-name value: $(body.repository.name) - name: git-revision value: $(body.head_commit.id)The API version of the TriggerBindingresource. In this example,v1beta1.Specifies the type of Kubernetes object. In this example, TriggerBinding.Unique name to identify the TriggerBindingresource.List of parameters which will be extracted from the received event payload and passed to the TriggerTemplateresource. In this example, the Git repository URL, name, and revision are extracted from the body of the event payload.The
TriggerTemplateresource acts as a standard for the way resources must be created. It specifies the way parameterized data from theTriggerBindingresource should be used. A trigger template receives input from the trigger binding, and then performs a series of actions that results in creation of new pipeline resources, and initiation of a new pipeline run.The following example shows a code snippet of a
TriggerTemplateresource, which creates a pipeline run using the Git repository information received from theTriggerBindingresource you just created:apiVersion: triggers.tekton.dev/v1beta1 kind: TriggerTemplate metadata: name: vote-app spec: params: - name: git-repo-url description: The git repository url - name: git-revision description: The git revision default: pipelines-1.14 - name: git-repo-name description: The name of the deployment to be created / patched resourcetemplates: - apiVersion: tekton.dev/v1 kind: PipelineRun metadata: name: build-deploy-$(tt.params.git-repo-name)-$(uid) spec: taskRunTemplate: serviceAccountName: pipeline pipelineRef: name: build-and-deploy params: - name: deployment-name value: $(tt.params.git-repo-name) - name: git-url value: $(tt.params.git-repo-url) - name: git-revision value: $(tt.params.git-revision) - name: IMAGE value: image-registry.openshift-image-registry.svc:5000/pipelines-tutorial/$(tt.params.git-repo-name) workspaces: - name: shared-workspace volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 500MiThe API version of the TriggerTemplateresource. In this example,v1beta1.Specifies the type of Kubernetes object. In this example, TriggerTemplate.Unique name to identify the TriggerTemplateresource.Parameters supplied by the TriggerBindingresource.List of templates that specify the way resources must be created using the parameters received through the TriggerBindingorEventListenerresources.The
Triggerresource combines theTriggerBindingandTriggerTemplateresources, and optionally, theinterceptorsevent processor.Interceptors process all the events for a specific platform that runs before the
TriggerBindingresource. You can use interceptors to filter the payload, verify events, define and test trigger conditions, and implement other useful processing. Interceptors use secret for event verification. After the event data passes through an interceptor, it then goes to the trigger before you pass the payload data to the trigger binding. You can also use an interceptor to modify the behavior of the associated trigger referenced in theEventListenerspecification.The following example shows a code snippet of a
Triggerresource, namedvote-triggerthat connects theTriggerBindingandTriggerTemplateresources, and theinterceptorsevent processor.apiVersion: triggers.tekton.dev/v1beta1 kind: Trigger metadata: name: vote-trigger spec: taskRunTemplate: serviceAccountName: pipeline interceptors: - ref: name: "github" params: - name: "secretRef" value: secretName: github-secret secretKey: secretToken - name: "eventTypes" value: ["push"] bindings: - ref: vote-app template: ref: vote-app --- apiVersion: v1 kind: Secret metadata: name: github-secret type: Opaque stringData: secretToken: "1234567"The API version of the Triggerresource. In this example,v1beta1.Specifies the type of Kubernetes object. In this example, Trigger.Unique name to identify the Triggerresource.Service account name to be used. Interceptor name to be referenced. In this example, github.Desired parameters to be specified. Name of the TriggerBindingresource to be connected to theTriggerTemplateresource.Name of the TriggerTemplateresource to be connected to theTriggerBindingresource.Secret to be used to verify events. The
EventListenerresource provides an endpoint, or an event sink, that listens for incoming HTTP-based events with a JSON payload. It extracts event parameters from eachTriggerBindingresource, and then processes this data to create Kubernetes resources as specified by the correspondingTriggerTemplateresource. TheEventListenerresource also performs lightweight event processing or basic filtering on the payload using eventinterceptors, which identify the type of payload and optionally modify it. Currently, pipeline triggers support five types of interceptors: Webhook Interceptors, GitHub Interceptors, GitLab Interceptors, Bitbucket Interceptors, and Common Expression Language (CEL) Interceptors.The following example shows an
EventListenerresource, which references theTriggerresource namedvote-trigger.apiVersion: triggers.tekton.dev/v1beta1 kind: EventListener metadata: name: vote-app spec: taskRunTemplate: serviceAccountName: pipeline triggers: - triggerRef: vote-triggerThe API version of the EventListenerresource. In this example,v1beta1.Specifies the type of Kubernetes object. In this example, EventListener.Unique name to identify the EventListenerresource.Service account name to be used. Name of the Triggerresource referenced by theEventListenerresource.
Additional resources
For information on installing OpenShift Pipelines, see Installing OpenShift Pipelines.
For more details on creating custom CI/CD solutions, see Creating CI/CD solutions for applications using OpenShift Pipelines.
For more details on re-encrypt TLS termination, see Re-encryption Termination.
For more details on secured routes, see the Secured routes section.
Comments
Post a Comment