Pipeline Stages and Jobs

Mitch Mommers01/06/2026About 9 min

Pipeline Stages and Jobs

Understanding how pipelines organize work is essential for building efficient CI/CD workflows. This page explains pipelines, stages, jobs, steps, and how they execute in Azure Pipelines.

Pipeline hierarchy

Azure Pipelines, like most CI/CD systems, organizes work in a hierarchy:

Pipeline: the complete automated workflow
Stage: a major phase in the workflow, such as validate, build, or deploy
Job: a group of steps that run on an agent
Step: a single task or script inside a job

In short:

Pipeline → Stages → Jobs → Steps

1. Pipeline

The pipeline is the complete automated workflow, from trigger to completion.

A pipeline can include stages for validation, building, deployment to development, post-deployment testing, and deployment to test or production.

Example:

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main

stages:
  - stage: Validate
    # ...
  - stage: Build
    # ...
  - stage: Deploy
    # ...

The pipeline above starts when changes are pushed to the main branch. It then runs the configured stages.

2. Stage

A stage is a major phase of the pipeline. Stages usually represent meaningful parts of the delivery process.

Common stages include:

Validate: run quality checks and security scans
Build: compile code and create artifacts
Deploy Dev: deploy to a development environment
Deploy Test: deploy to a test environment
Deploy Prod: deploy to production

By default, stages run sequentially. This means that the next stage waits for the previous stage to finish successfully.

Key characteristics:

Stages run in order by default.
A stage usually runs only if the previous stage succeeded.
You can control stage order with dependsOn.
You can control whether a stage runs with condition.

Example:

stages:
  - stage: Validate
    jobs:
      - job: Tests
        steps:
          - script: npm test
            displayName: 'Run tests'

  - stage: Build
    dependsOn: Validate
    condition: succeeded()
    jobs:
      - job: BuildArtifact
        steps:
          - script: npm run build
            displayName: 'Build application'

In this example, the Build stage waits for Validate and only runs if Validate succeeds.

3. Job

A job is a group of steps that run on an agent. An agent is the machine that executes the work.

Jobs in the same stage run in parallel by default, unless you configure dependencies between them. In practice, parallel execution also depends on the number of available agents and parallel jobs in your Azure DevOps organization.

Key characteristics:

A job runs on an agent, such as Ubuntu, Windows, or macOS.
Jobs in the same stage can run in parallel.
Each Microsoft-hosted job gets a fresh virtual machine.
Jobs can depend on other jobs by using dependsOn.

Example:

- stage: Validate
  jobs:
    - job: QualityGates
      pool:
        vmImage: 'ubuntu-latest'
      steps:
        - script: npm test
          displayName: 'Run tests'

    - job: SecurityScans
      pool:
        vmImage: 'ubuntu-latest'
      steps:
        - script: semgrep scan
          displayName: 'Run Semgrep scan'

In this example, QualityGates and SecurityScans can run in parallel because neither job depends on the other.

4. Step

A step is a single task or script inside a job. Steps run sequentially within the job.

Common step types include:

Script: run a shell command, such as bash, pwsh, or script
Task: use a built-in Azure Pipelines task, such as NodeTool@0
Checkout: clone the repository

Example:

steps:
  - task: NodeTool@0
    displayName: 'Install Node.js'
    inputs:
      versionSpec: '22.x'

  - script: npm ci
    displayName: 'Install dependencies'

  - script: npm test
    displayName: 'Run tests'

These steps run in order:

Install Node.js
Install dependencies
Run tests

Execution flow

The hierarchy also determines how long a pipeline takes to run.

Stages are sequential by default

Total time: 5 + 3 + 2 = 10 minutes

Stages are often sequential because they represent phases with dependencies:

You usually validate before building.
You usually build before deploying.
You usually deploy before running post-deployment tests.

Jobs can run in parallel

Total time: max(5, 3, 4) = 5 minutes

The stage starts all independent jobs at the same time. The stage only completes when all jobs in that stage have finished. In this example, the longest job takes 5 minutes, so the whole stage takes 5 minutes.

This assumes enough agents and parallel jobs are available in Azure DevOps.

Unit tests
Type checking
Dependency scanning
Infrastructure-as-code scanning

This gives faster feedback to the team.

Steps are sequential

Total time: 30s + 2min + 10s = 2min 40s

Steps are sequential because they often depend on each other:

You cannot run tests before installing dependencies.
You cannot publish test results before running the tests.
You cannot deploy an artifact before creating it.

Real-world example

The Day 3 pipeline uses multiple stages:

stages:
  # STAGE 1: Validate
  - stage: Validate
    jobs:
      - job: QualityGates
      - job: SecurityCodeAnalysis
      - job: SecurityDependencyScanning
      - job: SecurityIaCScanning

  # STAGE 2: Build
  - stage: Build
    dependsOn: Validate
    jobs:
      - job: BuildArtifact

  # STAGE 3: Deploy Dev
  - stage: DeployDev
    dependsOn: Build
    jobs:
      - deployment: DeployDevEnvironment

  # STAGE 4: Post-deployment tests
  - stage: PostDeploymentTests
    dependsOn: DeployDev
    jobs:
      - job: ZAPBaseline

  # STAGE 5: Deploy Test
  - stage: DeployTest
    dependsOn: PostDeploymentTests
    jobs:
      - deployment: DeployTestEnvironment

Execution timeline

Time    Stage           Jobs running
0:00    Validate        QualityGates
                        SecurityCodeAnalysis
                        SecurityDependencyScanning
                        SecurityIaCScanning
                        all 4 jobs can run in parallel

4:00    Validate ends   longest validation job took 4 minutes

4:00    Build           BuildArtifact

6:00    Build ends

6:00    DeployDev       DeployDevEnvironment

9:00    DeployDev ends

9:00    PostDeploy      ZAPBaseline

14:00   PostDeploy ends

14:00   DeployTest      DeployTestEnvironment

17:00   Pipeline complete

Total time: 17 minutes

If all jobs ran sequentially, the pipeline would take longer:

QualityGates:                  3 min
SecurityCodeAnalysis:          2 min
SecurityDependencyScanning:    4 min
SecurityIaCScanning:           2 min
BuildArtifact:                 2 min
DeployDevEnvironment:          3 min
ZAPBaseline:                   5 min
DeployTestEnvironment:         3 min
-------------------------------------
Total:                        24 min

Parallelism saves 7 minutes, which is about 29% faster.

The exact savings depend on available agents, available parallel jobs, and how long each job actually takes.

Stage dependencies and conditions

You can control execution flow with dependsOn and condition.

Simple dependency

Stage B waits for Stage A:

- stage: A
  jobs:
    - job: JobA
      steps:
        - script: echo "Stage A"

- stage: B
  dependsOn: A
  jobs:
    - job: JobB
      steps:
        - script: echo "Stage B"

Multiple dependencies

Stage C waits for both Stage A and Stage B:

- stage: A
  jobs:
    - job: JobA
      steps:
        - script: echo "Stage A"

- stage: B
  jobs:
    - job: JobB
      steps:
        - script: echo "Stage B"

- stage: C
  dependsOn:
    - A
    - B
  jobs:
    - job: JobC
      steps:
        - script: echo "Stage C"

In this example, Stage A and Stage B can run in parallel. Stage C waits until both are finished.

Conditional execution

You can use conditions to decide whether a stage, job, or step should run.

- stage: Deploy
  dependsOn: Build
  condition: succeeded()
  jobs:
    - deployment: DeployApp
      environment: dev
      strategy:
        runOnce:
          deploy:
            steps:
              - script: ./deploy.sh

Common conditions include:

Condition	Meaning
`succeeded()`	Run only if previous dependencies succeeded
`failed()`	Run only if previous dependencies failed
`always()`	Run regardless of previous result
`and(...)`	Combine multiple conditions
`eq(...)`	Compare values

Example with branch logic:

condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))

This runs only when previous dependencies succeeded and the pipeline is running on the main branch.

Fan-out pattern

A fan-out pattern runs multiple independent stages or jobs after one shared dependency.

- stage: Build
  jobs:
    - job: BuildArtifact
      steps:
        - script: npm run build

- stage: DeployDev
  dependsOn: Build
  jobs:
    - deployment: DevDeploy
      environment: dev
      strategy:
        runOnce:
          deploy:
            steps:
              - script: ./deploy-dev.sh

- stage: DeployTest
  dependsOn: Build
  jobs:
    - deployment: TestDeploy
      environment: test
      strategy:
        runOnce:
          deploy:
            steps:
              - script: ./deploy-test.sh

In this example, Build runs first. After that, DeployDev and DeployTest can run in parallel because both depend on Build, but not on each other.

Job types

Azure Pipelines supports different job types. The two most common types are regular jobs and deployment jobs.

Regular job

A regular job runs steps on an agent.

jobs:
  - job: MyJob
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - script: echo "Hello"

Use regular jobs for tasks such as:

Installing dependencies
Running tests
Running security scans
Building artifacts
Publishing results

Deployment job

A deployment job is a special job type for deployments to environments.

jobs:
  - deployment: DeployToProduction
    environment: production
    strategy:
      runOnce:
        deploy:
          steps:
            - script: ./deploy-app.sh

Benefits:

Tracks deployment history in Azure DevOps environments
Makes deployments visible in the Environments dashboard
Supports deployment strategies such as runOnce, rolling, and canary
Can be combined with environment checks and approvals

Important: deployment jobs do not automatically create rollback behavior. If rollback is needed, you must design and implement the rollback process yourself.

Job dependencies

Jobs can depend on other jobs in the same stage.

- stage: Test
  jobs:
    - job: UnitTests
      steps:
        - script: npm run test:unit

    - job: IntegrationTests
      dependsOn: UnitTests
      condition: succeeded()
      steps:
        - script: npm run test:integration

In this example, IntegrationTests only starts after UnitTests has completed successfully.

Use this pattern when one job should only run after another job has produced a successful result.

Agent pools

Jobs run on agents. An agent is the machine that executes the job.

You choose the agent with the pool setting.

Microsoft-hosted agents

Microsoft-hosted agents are managed by Microsoft.

pool:
  vmImage: 'ubuntu-latest'

Common hosted image labels include:

ubuntu-latest
windows-latest
macOS-latest

These labels are maintained by Microsoft and can change over time. For example, ubuntu-latest should be treated as “the current Microsoft-hosted Ubuntu image,” not as a fixed Ubuntu version.

Benefits:

No machine maintenance required
Fresh environment for each job
Multiple operating systems available

Limitations:

Limited customization
Hosted image contents can change over time
Network access may require extra configuration
Free-tier minutes and parallelism are limited

Self-hosted agents

Self-hosted agents are machines that you manage yourself.

pool:
  name: 'MyAgentPool'

Use cases:

You need specific tools or versions installed.
Your pipeline needs access to private networks.
You want more control over performance, storage, or installed software.
You want to avoid hosted-agent minute limits.

Trade-off: with self-hosted agents, you are responsible for maintenance, updates, security, and capacity.

Step types and tasks

Steps are the individual actions inside a job.

Script step

A script step runs shell commands.

- script: |
    echo "Installing dependencies"
    npm ci
    npm test
  displayName: 'Install and test'

Azure Pipelines also supports shell-specific shortcuts:

- bash: npm test
  displayName: 'Run tests with Bash'

- pwsh: Get-Date
  displayName: 'Run PowerShell command'

Task step

A task step uses a built-in Azure Pipelines task.

- task: NodeTool@0
  inputs:
    versionSpec: '22.x'
  displayName: 'Install Node.js'

Common tasks include:

Task	Purpose
`NodeTool@0`	Install Node.js
`UseDotNet@2`	Install .NET SDK
`Docker@2`	Build or push Docker images
`AzureWebApp@1`	Deploy to Azure App Service
`PublishTestResults@2`	Publish test results
`PublishPipelineArtifact@1`	Publish pipeline artifacts
`Cache@2`	Cache dependencies between pipeline runs

Checkout step

The checkout step clones the repository.

Azure Pipelines checks out the current repository by default, but you can customize the behavior:

- checkout: self
  clean: true
  fetchDepth: 1

Useful options:

clean: true: clean the working directory before checkout
fetchDepth: 1: perform a shallow clone for faster checkout

Download artifact step

A later stage can download artifacts created earlier in the pipeline.

- download: current
  artifact: 'backend-app'
  displayName: 'Download build artifact'

This is common in deployment stages. The build stage creates an artifact, and the deployment stage downloads and deploys it.

Variables and parameters

Variables let you reuse values in the pipeline.

Pipeline-level variables

Pipeline-level variables are defined at the top of the YAML file.

variables:
  nodeVersion: '22.x'
  buildConfiguration: 'Release'

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - script: echo "Node version: $(nodeVersion)"

Stage-level variables

Stage-level variables are available inside one stage.

- stage: Deploy
  variables:
    environmentName: 'production'
  jobs:
    - job: DeployJob
      steps:
        - script: echo "Deploying to $(environmentName)"

Job-level variables

Job-level variables are available inside one job.

- job: BuildJob
  variables:
    outputPath: '$(Build.ArtifactStagingDirectory)'
  steps:
    - script: echo "Output to $(outputPath)"

Built-in variables

Azure Pipelines provides many built-in variables.

Variable	Meaning
`$(Build.BuildId)`	Unique build number
`$(Build.SourceVersion)`	Git commit SHA
`$(Build.SourceBranch)`	Branch name, for example `refs/heads/main`
`$(Build.ArtifactStagingDirectory)`	Directory for staging build artifacts
`$(Agent.OS)`	Operating system of the agent, such as `Linux`, `Windows`, or `Darwin`

Built-in variables are useful for logging, naming artifacts, conditional logic, and deployments.

Best practices

Organize stages by delivery phase

Use stages to show the main delivery flow clearly.

stages:
  - stage: Validate
  - stage: Build
  - stage: DeployDev
  - stage: DeployTest
  - stage: DeployProd

Clear stages make the pipeline easier to understand and debug.

Parallelize independent jobs

If jobs do not depend on each other, let them run in parallel.

- stage: Validate
  jobs:
    - job: TypeCheck
    - job: Tests
    - job: SecurityScan

This provides faster feedback.

Fail fast

Put fast checks before slow checks.

steps:
  - script: npm run typecheck
    displayName: 'Run type check'

  - script: npm test
    displayName: 'Run tests'

  - script: npm run e2e-tests
    displayName: 'Run end-to-end tests'

If type checking fails, the job stops before spending time on slower tests.

Use meaningful display names

Use display names that explain the intent of each step.

- script: npm test
  displayName: 'Run unit tests'

This makes pipeline logs easier to read.

Cache dependencies

Caching can reduce pipeline time by reusing downloaded dependencies.

- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    path: '$(Pipeline.Workspace)/.npm'
  displayName: 'Cache npm packages'

Caching is useful for package managers such as npm, Maven, NuGet, or pip.

Avoid monolithic jobs

A single large job is harder to read, debug, and reuse.

Avoid this:

- job: Everything
  steps:
    - script: npm run build
    - script: npm test
    - script: semgrep scan
    - script: ./deploy.sh

Prefer this:

- stage: Validate
  jobs:
    - job: Tests
    - job: SecurityScan

- stage: Build
  jobs:
    - job: BuildArtifact

- stage: Deploy
  jobs:
    - deployment: DeployApp

Separation improves clarity and makes the pipeline easier to maintain.

Use deployment jobs for environment deployments

Use deployment jobs when deploying to named environments.

- deployment: DeployToTest
  environment: test
  strategy:
    runOnce:
      deploy:
        steps:
          - script: ./deploy-test.sh

This gives better visibility into what was deployed, when it was deployed, and to which environment.

Summary

Pipeline hierarchy:

Pipeline: complete workflow
Stage: major phase, usually sequential
Job: group of steps running on an agent, parallel by default when independent
Step: individual task or script, sequential within a job

Key concepts:

Stages represent major phases such as validate, build, and deploy.
Jobs inside a stage can run in parallel when they do not depend on each other.
Steps inside a job run in order.
Use dependsOn and condition to control execution flow.
Parallel jobs require available agents and sufficient Azure DevOps parallel-job capacity.

Best practices:

Fail fast by running cheap checks before expensive checks.
Parallelize independent work.
Use deployment jobs for environment-based deployments.
Cache dependencies to speed up repeated pipeline runs.
Avoid large monolithic jobs when separate stages or jobs would be clearer.

In the labs, you will build a multi-stage pipeline that uses these concepts to create a secure and efficient delivery workflow.