Integrating Docker into CI/CD Pipelines

Introduction

Continuous Integration and Continuous Deployment/Delivery (CI/CD) pipelines are essential for modern software development. They automate testing, building, and deploying applications, enabling teams to deliver features and fixes faster with higher reliability. Docker plays a pivotal role in CI/CD by providing consistent, reproducible environments, isolating dependencies, and simplifying deployment.

In this lesson, you'll learn how to integrate Docker into CI/CD pipelines, examine real-world workflows, and explore best practices and pitfalls to avoid.

Why Use Docker in CI/CD?

Docker enhances CI/CD in several ways:

Consistency: Ensures the same environment across development, testing, and production.
Isolation: Prevents dependency conflicts and "works on my machine" issues.
Speed: Enables faster pipeline execution by leveraging image caching and parallelization.
Portability: Makes it easy to run builds and tests anywhere Docker is available.

Docker in a Typical CI/CD Pipeline

A typical pipeline with Docker includes:

Code Checkout: Retrieve source code from version control.
Build: Compile code and build Docker images.
Test: Run unit/integration tests within containers.
Push: Upload built images to a Docker registry.
Deploy: Pull and run the image in target environments (staging, production).

Let's see how these steps are implemented in popular CI/CD platforms.

Example: Docker with GitHub Actions

GitHub Actions is a popular CI/CD platform. Below is a sample workflow file (.github/workflows/docker-ci.yml) that builds, tests, and pushes a Docker image:

name: CI with Docker

on:
  push:
    branches: [ "main" ]

jobs:
  build-test-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Log in to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Tests
        run: docker run --rm myapp:${{ github.sha }} npm test

      - name: Push image to DockerHub
        run: docker push myapp:${{ github.sha }}

Key Steps:

Sets up Docker and authenticates with DockerHub.
Builds an image with the current commit SHA tag.
Runs tests inside the container.
Pushes the image to DockerHub for deployment.

Example: Docker in GitLab CI

Here's a simplified .gitlab-ci.yml example:

stages:
  - build
  - test
  - push

variables:
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  script:
    - docker build -t $IMAGE_TAG .

test:
  stage: test
  script:
    - docker run --rm $IMAGE_TAG pytest

push:
  stage: push
  script:
    - echo $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
    - docker push $IMAGE_TAG
  only:
    - main

Explanation:

Uses built-in GitLab CI/CD environment variables.
Builds, tests, and pushes a Docker image for every commit to main.

Running Tests in Containers

You can run tests using Docker containers to ensure environment consistency. For example, with a Node.js project:

# Build the image
docker build -t myapp:test .

# Run tests
docker run --rm myapp:test npm test

This approach ensures that your tests run in the same environment as your application.

Deploying with Docker in CI/CD

After pushing an image to a registry, deployment steps might include:

Using docker-compose to pull and start new containers on a server.
Updating Kubernetes deployments to reference the new image tag.

Example: Updating a Docker Compose app via SSH

- name: Deploy via SSH
  uses: appleboy/ssh-action@v0.1.10
  with:
    host: ${{ secrets.SERVER_HOST }}
    username: ${{ secrets.SERVER_USER }}
    key: ${{ secrets.SERVER_SSH_KEY }}
    script: |
      docker pull myapp:${{ github.sha }}
      docker-compose up -d

Use Cases

Microservices: Build and test multiple services in parallel using Docker Compose in CI.
Legacy app modernization: Wrap existing apps in Docker images for easier integration into pipelines.
Automated security scanning: Run vulnerability scanners (like trivy or clair) as pipeline steps.
Multi-architecture builds: Use Docker Buildx for ARM/x86 images in the pipeline.

Common Mistakes and Pitfalls

Not cleaning up containers/images: Can lead to disk space exhaustion in CI runners.
Hardcoding secrets: Always use CI/CD secret management; never store registry credentials in source code.
Tagging every image as latest: Use unique tags (e.g., commit SHA) to avoid confusion and deployment errors.
Ignoring build cache: Proper cache usage can significantly speed up builds.
Running tests outside containers: Reduces consistency and may miss environment-specific bugs.

Summary

Integrating Docker into your CI/CD pipelines unlocks the full potential of containerization: faster, more reliable, and more portable deployments. Popular CI/CD tools like GitHub Actions and GitLab CI provide first-class Docker support, allowing you to build, test, and deploy applications with confidence. Remember to avoid common pitfalls, leverage secrets management, and use proper image tagging strategies.

Quiz

What are two major benefits of running tests inside Docker containers during CI?
- Answer: Ensures consistent environments and prevents dependency conflicts.
Why is it a bad practice to tag every built Docker image as latest in a CI/CD pipeline?
- Answer: It can cause confusion and accidental deployments of the wrong version, as latest does not uniquely identify a build.
Name one way to securely provide Docker registry credentials in a CI/CD pipeline.
- Answer: Use the CI/CD platform's built-in secrets management (such as GitHub Secrets or GitLab CI/CD Variables).
What command would you use to run unit tests inside a container for an image named myapp:test?
- Answer: docker run --rm myapp:test <test-command> (e.g., docker run --rm myapp:test npm test).
List one common mistake when using Docker in CI/CD pipelines and how to avoid it.
- Answer: Not cleaning up unused containers or images, which can fill up disk space; avoid this by using --rm for containers and configuring periodic cleanup jobs.

Introduction​

Why Use Docker in CI/CD?​

Docker in a Typical CI/CD Pipeline​

Example: Docker with GitHub Actions​

Example: Docker in GitLab CI​

Running Tests in Containers​

Deploying with Docker in CI/CD​

Use Cases​

Common Mistakes and Pitfalls​

Summary​

Quiz​