CI/CD: Complete Guide to Continuous Integration and Delivery

 

What Is CI/CD?

Continuous Integration (CI) and Continuous Delivery/Deployment (CD), collectively known as CI/CD, represent pivotal practices in modern software engineering. CI is the process where developers regularly integrate their code changes into a central repository. Each integration is then automatically tested and verified, promoting high-quality code and early bug detection. On the other hand, CD takes this a step further by automating the delivery of these tested code changes to predefined infrastructure environments, ensuring seamless and reliable software updates. With this automated build, testing, and deployment process, CI/CD practices enable teams to release software faster and more reliably, making it a cornerstone of DevOps culture.

A CI/CD pipeline compiles incremental code changes made by developers and packages them into software artifacts. Automated testing verifies the integrity and functionality of the software, and automated deployment services make it immediately available to end users. The goal is to enable early detection of defects, increase productivity, and shorten release cycles.

This process contrasts with the traditional approach to software development—consolidating multiple small software updates into one large release, thoroughly testing it, and only then deploying it. CI/CD pipelines support the agile concept of development in small iterations, enabling teams to deliver value to customers faster, and create a rapid feedback loop for developers.

What Are the Differences Between Continuous Integration, Continuous Delivery, and Continuous Deployment?

Continuous Integration

In the traditional software development process, multiple developers produce code, and only towards the end of a release do they consolidate their work. This caused many bugs and issues, which could only be identified and resolved after a long testing phase. Until all those issues were resolved, the software could not be released. This hurt software quality, and meant that teams could typically only release new versions once or twice a year.

Continuous Integration (CI) was designed to solve this problem and support agile development processes. CI means that any changes developers make to their code are immediately integrated into the master branch of the software project. The CI system automatically runs tests to catch quality issues, and developers get quick feedback and can fix issues immediately. Developers often commit to the master branch or work on a short-lived feature branch, and a feature is not considered complete until it is integrated with other code changes in the master branch.

In a CI process, a build server is responsible for taking new code changes, running automated tests using multiple tools, integrating the code into the master branch, and generating a build—a new version of software artifacts needed to deploy the software.

CI greatly improves the quality and speed of software development. Teams can create more features that provide value to users, and many organizations now release software every week, every day, or multiple times a day.

Continuous Delivery

Traditionally, deploying new software versions has been a large, complex and risky task. After the new version was tested, the operations team was tasked with deploying it into production. Depending on the size of the software it could take hours, days or weeks, requires detailed checklists and many manual steps, and special expertise. Deployments often failed and required developer workarounds or urgent assistance.

There are many problems with this traditional approach—it is stressful for the team, expensive and risky for the organization, and causes bugs and downtime in production environments.

Continuous Delivery (CD, also known as CDel) aims to solve these problems through automation. The CD approach allows teams to package software and deploy it to production environments with the push of a button. The basic principle of CD is that any change to a software project can be deployed to a production environment immediately, without any special effort.

After the CI system consolidates the new changes and creates a new build, the CD system packages the new version, deploys it to a test environment, automatically evaluates its behavior, and pushes it to the production environment. This last step can be manually approved, but no manual action is required to deploy the new version to production.

Implementing CD requires automating the entire software development lifecycle, including build, test, environment setup, and deployment. All artifacts must reside in a source code repository, and an automated mechanism is required to create and update the environment.

A true CD pipeline has great advantages. It allows development teams to deliver value to customers quickly and create truly agile development processes.

Continuous Deployment

Continuous Deployment (CDep) goes one step further than continuous delivery. All changes going through all stages of the production pipeline undergo automated tests, and if these tests pass, they are immediately deployed to production and exposed to customers.

Continuous deployment puts an end to release dates, and is a great way to speed up the customer feedback loop and reduce stress on the team. Developers can focus on building the software and see it running in production minutes after completion. 

Continuous deployment can be difficult to implement. It requires seamless automation at all stages of the process, robust automated testing suites, and a culture of “continuous everything” that enables detection and rapid response to production issues.

How Does CI/CD Relate to DevOps?

DevOps promotes better collaboration and communication between development (Dev) and operations (Ops) teams. It often requires changing various aspects of the development lifecycle, including job roles, tools, best practices, and automating the lifecycle. 

DevOps typically involves the following: 

• Adopting automation, programmable infrastructure deployment and maintenance, and iterative software development. 
• Establishing cross-functional teams while facilitating a culture change to build trust between these previously disparate teams. 
• Aligning technologies to business requirements. 

CI/CD supports the efforts of DevOps teams. It enables teams to implement automation across the development lifecycle and rapidly validate and deliver applications to end-users. Here is how it works:

• Continuous integration tools help initialize processes, ensuring developers can build, test, and validate code within a shared repository without manual work. 
• Continuous delivery tools extend these automated steps to production testing and configuration for release management. 
• Continuous deployment tools automatically invoke tests, handling configurations, provisioning, monitoring, and rollbacks.

What Are the Stages of a CI/CD Pipeline?

The CI/CD pipeline performs continuous integration, delivery, and deployment in four phases—source, build, test, and deploy. 

Source

Creating source code is the first phase in a CI/CD pipeline. During this phase, developers translate requirements into functional algorithms, features, and behaviors. Tools often vary, depending on the project, the project’s language, and other variables. As a result, there is no uniform source creation pipeline. 

A source code creation pipeline may incorporate any of the following:

• A programming framework, such as Java, .NET, C#, or PHP.
• An integrated development environment (IDE) that supports the programming language chosen for the project. 
• Code-checking tools, such as vulnerability scanners, basic error detection, and tools verifying adherence to coding standards. 
• Code repositories and version control systems, such as Git.

Build

The build phase involves pulling source code from a repository, establishing links to libraries, dependencies, and modules, and building these components into an executable (.exe) file. It typically requires tools that can generate execution logs, denote errors to correct and investigate, and notify developers once a build is completed.

Build tools vary according to the programming language. Some scenarios may require a specific build tool, while others can employ the same IDE for both source and build phases. A build phase may use additional tools to translate an executable file into a deployable or packaged execution environment, such as a virtual machine (VM) or a Docker container.

Test

During the source code creation phase, the code undergoes static testing. The completed build enters the next CI/CD phase to undergo dynamic testing, including: 

• Basic functional or unit testing—helps validate new features work as intended.
• Regression testing—helps ensure changes do not break previously working features.

In addition to functional and regression tests, the build undergoes tests that verify integration, performance, and user acceptance. If errors occur during the testing phase, the process loops these results back to developers for analysis and remediation. Since builds undergo many tests, developers employ automated testing to minimize human error and improve productivity.

Deploy

After a build passes the testing phase, it becomes a candidate for deployment. There are two main ways to deploy the build, including:

• Continuous delivery—the build is sent to human staff for approval and then deployed. For example, new versions are automatically deployed to a test environment, but promotion to production is gated by a manual approval or merge request.
• Continuous deployment—the pipeline automatically deploys the build to testing, staging, and production environments, assuming it passes all relevant tests, with no manual approvals.

A typical deployment phase creates a deployment environment and moves the build to a deployment target, like a server. You can automate these steps with scripts or workflows in automation tools. Most deployments also integrate with error reporting and ticketing tools to detect unexpected errors post-deployment and alert developers.

Benefits of Kubernetes for CI/CD Pipelines

The secret to the success of a CI/CD pipeline is ensuring that application updates are performed quickly and in an automated manner. Teams typically face the following challenges when adopting CI/CD:

• Manual steps in the release process—Many CI/CD processes still use manual testing and deployment steps. This can cause delays and affect production schedules. Manual CI/CD processes can cause code merge conflicts and increase customer wait times for patches and updates.
• Downtime risk—manual infrastructure management processes can be a headache for DevOps teams because they create the risk of downtime. For example, unexpected traffic spikes that exceed capacity can cause downtime and require manual steps to restore applications.
• Inefficient resource utilization—applications are often deployed on servers in an inefficient way. This means organizations have to pay more for capacity. As applications are added, scaled up and down, it can be difficult to efficiently use available hardware resources. This is true whether the application is running in the cloud or on-premises.

Kubernetes can solve all three of these problems. It reduces the time and effort required to develop and deploy applications in a CI/CD pipeline. Its efficient resource management model increases hardware utilization, automates management processes, and reduces disruptions that negatively impact customers. Specifically, Kubernetes can:

• Cluster Management – Kubernetes takes the best practices for all previous clustering solutions and packages them in a vendor agnostic way. It comes bundled with several critical components such as schedulers and resource managers and contains plugin mechanisms for storage, networking, secrets etc. Writing distributed applications with Kubernetes is much easier compared to legacy clustering solutions as the environment it offers is standardized and without any proprietary mechanisms to closed systems
• Orchestrate deployment and provisioning—coordinating provisioning activities and simplifying deployment. Kubernetes handles hardware and storage resource configuration, software deployment, scalability, and health monitoring, and is fully customizable for specific needs.
• Declarative constructs—codifying the final state of the desired environment or application in simple, human readable code. This makes it possible to recover faster from downtime and production issues, better control scaling, and streamline disaster recovery processes.

CI/CD in the Cloud

CI/CD in the cloud refers to the practice of using cloud-based services to perform Continuous Integration and Continuous Delivery/Deployment (CI/CD) of software. This enables developers to build, test, and deploy their software faster and more efficiently by leveraging the scalability, flexibility, and cost-effectiveness of the cloud.

Cloud providers, such as AWS, Azure, Google Cloud, and others, offer a wide range of services that can be used to implement CI/CD in the cloud. These services include:

• Cloud-based source control: Providers offer cloud-based source control services, such as GitHub, GitLab, and Bitbucket, which can be used to store and manage code.
• Build and test automation: Services such as AWS CodeBuild, Azure DevOps, and Google Cloud Build can be used to automate the building and testing of code.
• Containerization and orchestration: Services such as AWS Elastic Container Service, Azure Container Instances, and Google Kubernetes Engine can be used to containerize, orchestrate, and migrate applications, which makes it easier to deploy and manage them in the cloud.
• Deployment automation: Services such as AWS CodeDeploy, Azure DevOps, and Google Cloud Deployment Manager can be used to automate the deployment of code to various environments, including development, staging, and production environments.
• Monitoring and logging: Cloud providers offer various services such as AWS CloudWatch, Azure Monitor, and Google Stackdriver that can be used to monitor and log the performance and status of cloud-based applications and infrastructure.

By using these services, developers can easily set up, configure and manage a CI/CD pipeline in the cloud, which will increase the speed of delivery of new features and bug fixes to customers and also help to detect and fix problems early in the development cycle.

CI/CD Tools

Here is a brief review of popular CI/CD tools.

Continuous Integration Tools

Popular CI tools include Codefresh, Bitbucket Pipelines, Jenkins, CircleCI, Bamboo, and GitLab CI.

Codefresh

Codefresh is a comprehensive GitOps continuous integration toolset designed for Kubernetes and modern applications. It is built from the ground up for flexibility and scalability around Argo Workflows and Argo Events. It takes the best of the open source toolset and provides essential enterprise features like a unified user interface, a single pane for cloud-wide management, security validated enterprise-grade runtime, end-to-end auditability, and cross-application single sign-on.

Bitbucket Pipelines

Bitbucket Pipelines is a CI tool that integrates directly into Bitbucket, a cloud-based source control system. It lets you manage pipelines as code and deploy your projects to production via CD tools. You can use Bitbucket pipelines to create pipeline definitions and kick off builds.

Jenkins

Jenkins is an open source automation tool that provides plugins to help develop, deploy, and deliver software. It is a server that lets developers distribute tasks across various machines and perform distributed tests and deployments. The Jenkins Pipeline offers several plugins to facilitate the implementation of a continuous integration (CI) pipeline.

CircleCI

CircleCI is a CI tool that supports various container systems, delivery mechanisms, and version control systems like Github. CircleCI can run complex pipelines with caching, resource classes, and Docker layer caching. You can run this tool in the cloud and on-premises.

Bamboo

Bamboo is an automation server for continuous integration that can automatically build, test, integrate, and document source code to prepare apps for deployment. It offers a simple user interface for CI/CD and various features, including automated merging and built-in deployment support.

GitLab CI

GitLab CI is an open source CI tool. It lets you use the GitLab API to install and set up projects hosted on GitLab. GitLab CI can help you test and build projects and deploy your builds. It indicates areas that require improvement and lets you secure project data using confidential issues.

Continuous Delivery and Deployment Tools

Popular CD tools include Codefresh, Argo CD, GoCD, AWS CodePipeline, Azure Pipelines, and Spinnaker.

Codefresh

Codefresh is a modern GitOps software delivery solution powered by Argo with support for advanced deployments like canary, blue-green, and experimental releases. It provides comprehensive dashboards that offer visibility from code to cloud while integrating with your favorite tools. A centralized dashboard gives insight into deployments at scale while providing the security and support enterprises need.

Argo CD

Argo CD is a Kubernetes-native CD tool optimized for GitOps. It stores configuration in a Git repository and automatically applies it to Kubernetes clusters, making it easy to integrate with existing workflows. Argo CD can detect configuration drift, monitor application health, and roll back unwanted configuration changes. It also supports progressive delivery strategies like blue/green and canary deployment.

GoCD

GoCD is an open source CD tool that helps automate the entire build-test-release process, including code check-in and all the way to deployment. It works with Git, Subversion, Mercurial, TFVC (TFS), and Perforce, and has an open plugin ecosystem. It is deployed on-premises.

AWS CodePipeline

AWS CodePipeline is a cloud-based CD service that helps model, visualize, and automate software release steps and continuous changes. Notable features include release process automation, establishing a consistent release process, and viewing pipeline history details.

Azure Pipelines

Azure Pipelines is a cloud-based service that helps automatically build, test, and ship code to multiple targets, through a combination of CI and CD mechanisms. It supports many languages, including Python, JavaScript, and Go, most application types, including Node.js and C++, and targets such as virtual machines (VMs), containers, on-premises, and cloud platforms.

Spinnaker

Spinnaker is an open source CD platform for multi-cloud environments. It offers a pipeline management system and integrates with many cloud providers. Spinnaker provides a pipeline builder to automate releases, and lets you save and reuse existing pipelines as JSON files. It supports Kubernetes and integrates with tools like Prometheus, Datadog, and StackDriver.

GitHub Actions

GitHub Actions is a CI/CD tool from GitHub, the world’s most popular platform for hosting and collaborating on software projects. GitHub Actions allows developers to automate workflows directly from their GitHub repositories, making it convenient for teams already using GitHub.

GitHub Actions allows developers to create custom workflows using simple YAML syntax, and they can leverage a marketplace of community-contributed actions to extend their workflows.

Harness.io

Harness.io is a CI/CD platform that emphasizes simplicity of usage. It offers a visual interface for building and managing deployment pipelines and can automate various CI/CD processes, such as canary deployments and rollback decisions. Harness.io automatically verifies deployments in real-time to detect and mitigate issues before they impact end-users.

CI/CD Security Risks

Supply Chain Attacks

A supply chain attack is a cyber attack that targets weak links in an organization’s supply chain. A supply chain is the network of all individuals, organizations, resources, activities, and technologies involved in creating and selling software products.

Modern software applications rely heavily on dependencies to provide their core functionality. The software ecosystem relies heavily on CI/CD to publish source code and binaries to public repositories. This allows attackers to bypass standard security measures and directly attack the supply chain, infecting many applications and websites simultaneously.

Insecure System Configuration

A CI/CD environment consists of several systems from various vendors. To optimize CI/CD security, security teams must focus on the health and resilience of individual systems and the code and artifacts flowing through the pipeline.

Like any other system that stores and processes data, a CI/CD system includes a variety of security settings and configurations at the application, network, and infrastructure levels. These settings have a significant impact on the security posture of a CI/CD environment and its vulnerability to potential breaches. Attackers are on the lookout for ways to exploit potential CI/CD vulnerabilities and misconfigurations.

Insecure Code

The demand for rapid software development and delivery has increased the use of open source third-party integrations. Some teams may bring third-party integrations into their deployments without properly scanning the source code for security vulnerabilities. Such integrations could lead to vulnerabilities in the CI/CD pipeline. Developers may not follow code security best practices, increasing the attack surface. Common code vulnerabilities include user input vulnerabilities, buffer overflows, error handling errors, and serialization issues.

Exposure of Secrets

Automated processes are a key component of any DevOps infrastructure. CI/CD orchestration and configuration tools are increasingly being deployed into DevOps processes to automate processes and facilitate rapid deployment of software releases.

However, CI/CD tools make extensive use of secrets (like passwords and API access tokens). they access many sensitive resources, such as information from other applications and services, code repositories, and databases. The more secrets you have, the more difficult it is to securely store, transmit, and audit them.

Also, secrets are not only used for tool-to-tool authentication. In many cases, confidential information must be provided during the build and deployment process so that deployed resources can access it. This is especially important when deploying microservices using the auto-scaling capabilities of tools like Kubernetes.

DevSecOps and CI/CD

DevSecOps is a philosophy and organizational culture that adopts security practices in DevOps processes. It is also used to represent a continuous delivery, security-centric software development lifecycle (SDLC).

Historically, security was seen as a secondary part of DevOps workflows. Information security practices were applied at the end of the software development lifecycle (SDLC). However, discovering security breaches at the end of SDLC can be very frustrating and issues discovered at that stage are difficult and expensive to resolve. DevSecOps drives security engagement as an active part of the software development lifecycle (SDLC) from its earliest stages.

A typical DevOps pipeline includes stages such as planning, coding, building, testing, release, and deployment. DevSecOps enforces specific security checks at each stage of the DevOps pipeline:

• Planning—during the planning phase, you create a plan that determines when, where, and how to perform a security analysis and test your scenarios.
• Coding—using linting tools and Git controls to secure passwords and API keys.
• Building—using static application testing (SAST) tools to track defects in code before deployment to production. These tools are specific to a programming language.
• Testing—during application testing, using Dynamic Application Security Testing (DAST) tools to detect errors related to user authentication, authorization, SQL injection, and API endpoints.
• Release—leveraging security analysis tools for vulnerability scanning and penetration testing. These tools should be used immediately before deploying the application.
• Deployment—after completing the above tests, the secure build is sent to production for final deployment. Deployments should be monitored at runtime for undiscovered security threats or vulnerabilities.

CI/CD Best Practices

Here are a few best practices that can help you practice CI/CD more effectively.

Learn about these and other best practices in our detailed guide to CI/CD best practices

Build Only Once

Eliminate the practice of building the same source code multiple times. If you need to build, package, or bundle your software, you only need to perform this step once and promote binaries to the next stage of the pipeline. 

Most successful CI implementations include the build process as the first step in the CI/CD cycle, making sure that software is packaged in a clean environment. This eliminates human error and reduces the chance of overlooked artifacts or incorrect artifacts included by mistake. Also, any artifacts generated must be versioned and uploaded to Git, to ensure that every time they are needed in the process, the same version of the build is available.

Prioritize Automation Efforts

Organizations moving to automated processes often struggle to identify which processes to automate first. For example, it is useful to automate the code compilation process from scratch. It is a good idea to run automated smoke tests every time developers commit new code. Unit tests are usually automated first to reduce developer workload.

In most cases, you will automate functional testing before UI testing. Unlike UI tests, which change more frequently, functional tests do not require frequent updates to automation scripts. Consider all possible dependencies, assess their impact, and prioritize automation as appropriate.

Release Often

A commercial release is only possible if the software is release-ready and tested in a production-like environment. Therefore, it is best to add a step that deploys new versions to a realistic pre-production staging environment, or to the production environment itself alongside the current production version. 

The following are release strategies that can help you deploy software to staging and production environments with low risk:

• Canary deployment—release the new version for some users, test their response, and if it works well, roll it out to a larger population. If the test fails, roll back and repeat.
• Blue/Green Deployment—run the current and new version of the software in two identical production environments. Initially, the current version is live and the new version is idle. Then traffic is switched over from the current version to the environment containing the new version. This lets you test the new version on real user traffic, and if something goes wrong, you can immediately roll back to the current stable version.
• A/B Testing—A/B testing is a method used to test the functionality of an application, such as changes to the user experience. Two or more versions of the application, with small differences between them, are served to production users in parallel. Teams observe how users interact with each version, and when one version is deemed successful, it is rolled out to all users. 

Make the CI/CD Pipeline the Only Way to Deploy to Production

Investing in building a reliable, fast, and secure CI/CD pipeline gives you confidence in your build quality, but bypassing that process for any reason can hurt your efforts. Requests to circumvent the release process often occur because changes are minor or urgent—you should not give in to these requests.

Skipping automated tests runs the risk of production issues, but the problem does not end there. It is much more difficult to reproduce and debug problems, and trace them to specific build artifacts, because builds are not automatically deployed to test and production environments.

Even if at some point a team makes an exception and skips the CI/CD process, it is worth understanding the motive. Why was the request to skip the CI/CD pipeline made in the first place? Talk to key stakeholders and identify if the process seems slow or inflexible to them. You may need to make performance or process improvements to address those concerns. 

By remaining responsive to stakeholder requirements, and communicating the benefits of the CI/CD pipeline, you can convince stakeholders and avoid disrupting the CI/CD process due to urgent requests.

Clean Up Environments with Every Release

To get the most out of your testing process, it’s worth cleaning up pre-production environments before each deployment. 

If your environment has been running for a long time, it can be difficult to keep track of all configuration changes and updates applied—this is known as configuration drift. This means tests may not return the same results. Maintaining a static environment incurs maintenance costs, slows down testing, and delays the release process.

By using containers to host environments and run tests, you can easily start and destroy each newly deployed environment by scripting these steps using declarative configuration (for example, Kubernetes YAML files). Instantiating new containers before each deployment ensures consistency, and makes it easy to scale your environment to test multiple builds in parallel.

Taking CI/CD to the Next Level with Codefresh

Codefresh has made it our mission since 2014 to help teams accelerate their pace of innovation.  Our platform, powered by Argo, combines the best of the open-source with an enterprise-grade runtime allowing you to fully tap the power of Argo Workflows, Events, CD, and Rollouts. It provides teams with a unified GitOps experience to build, test, deploy, and scale their applications.