Before Lab 3.6: Artifact Promotion, Environment Gates, and Rollback

What You'll Learn

You've built the application once, scanned it, tested it, and deployed it to dev. Now you're ready to deploy to test and production.

Critical question: Are you deploying the exact same artifact that passed all quality and security gates, or are you rebuilding from source for each environment?

This lab focuses on:

• Build-once-promote-many: Compile, package, and scan once; deploy the same artifact everywhere
• Artifact traceability: Proving which commit, build, tests, and scans produced a deployed artifact
• Immutable artifacts: Preventing unauthorized changes between environments
• Deployment confidence: Knowing production runs the same bits that passed dev/test validation

---

The Problem with Rebuild-Per-Environment

Traditional Anti-Pattern

Many teams rebuild the application for each environment:

# Anti-pattern: Rebuild for each environment
- stage: DeployDev
  steps:
    - script: npm run build  # Build for dev
    - script: deploy to dev

- stage: DeployTest
  steps:
    - script: npm run build  # Rebuild for test (different result?)
    - script: deploy to test

- stage: DeployProd
  steps:
    - script: npm run build  # Rebuild for prod (different result?)
    - script: deploy to prod

Why This Is Dangerous

Risk	Description	Impact
Inconsistent builds	Source code or dependencies change between builds	Test passes, prod fails with different code
Untested artifacts	Production artifact was never tested in dev/test	Zero confidence in production deployment
Time drift	Dev deployed Monday, prod deployed Friday — different npm packages	Dependency vulnerabilities introduced mid-week
No traceability	Cannot prove prod runs the same code that passed security scans	Compliance audit failures

Real-world scenario: A critical security patch is tested in dev on Monday. By Friday, when deploying to prod, a new build pulls a vulnerable dependency version that was published Tuesday. Production is now vulnerable despite passing dev testing.

---

Build-Once-Promote-Many Pattern

The Correct Approach

# Correct: Build once, promote everywhere
- stage: Build
  steps:
    - script: npm run build
    - script: npm test
    - script: semgrep scan
    - script: trivy scan
    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: 'dist'
        artifactName: 'backend-app'

- stage: DeployDev
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'  # Same artifact
    - script: deploy to dev

- stage: DeployTest
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'  # Same artifact
    - script: deploy to test

- stage: DeployProd
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'  # Same artifact
    - script: deploy to prod

Key Principles

1. Build exactly once: Compile, transpile, bundle in a single Build stage
2. Scan the artifact: Trivy, Semgrep, and Checkov scan the built artifact or source before publishing
3. Publish to artifact storage: Azure Pipelines artifact storage, Azure Blob Storage, container registry
4. Download in deployment stages: Every environment downloads the same published artifact
5. Never rebuild: Deployment stages have no npm run build or equivalent

---

Artifact Metadata and Traceability

What Is Traceability?

When you look at a running production application, you should be able to answer:

• Which commit produced this deployment? (Git SHA)
• Which build created this artifact? (Build ID)
• Which tests passed before this was deployed? (Test results)
• Which security scans cleared this artifact? (Scan reports)

Embedding Metadata

In azure-pipelines.yml, the Build stage creates metadata files:

- stage: Build
  steps:
    - script: |
        echo "$(Build.BuildId)" > dist/BUILD_VERSION
        echo "$(Build.SourceVersion)" > dist/BUILD_COMMIT
        echo "$(Build.BuildNumber)" > dist/BUILD_NUMBER
      displayName: 'Embed Build Metadata'

    - script: npm run build
      displayName: 'Build Application'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: 'dist'
        artifactName: 'backend-app'

Result: The published artifact contains:

• BUILD_VERSION: Azure Pipelines build ID (e.g., 12345)
• BUILD_COMMIT: Git commit SHA (e.g., abc123def456)
• BUILD_NUMBER: Semantic version or build number (e.g., 1.2.3-20260531.1)

Figuur 0: Azure DevOps pipeline artifacts showing backend-app and metadata files

Figuur 0: Artifact metadata files (BUILD_COMMIT, BUILD_VERSION, BUILD_NUMBER) in file explorer

Verifying in Deployment

In deployment stages, you can verify the metadata:

- stage: DeployProd
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'
        downloadPath: '$(System.DefaultWorkingDirectory)/artifact'

    - script: |
        echo "Deploying artifact from commit: $(cat artifact/BUILD_COMMIT)"
        echo "Built in pipeline run: $(cat artifact/BUILD_VERSION)"
        echo "Build number: $(cat artifact/BUILD_NUMBER)"
      displayName: 'Verify Artifact Metadata'

Audit trail: Logs show exactly which commit/build is being deployed, enabling rollback and compliance reporting.

---

Artifact Storage Options

Azure Pipelines Artifact Storage

Built-in storage attached to the pipeline run:

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: 'dist'
    artifactName: 'backend-app'

Pros:

• Free, integrated with Azure DevOps
• Automatic retention policies
• Easy to download in later stages

Cons:

• Tied to pipeline run (deleted when pipeline is deleted)
• Not suitable for long-term artifact retention

Azure Container Registry (ACR)

For containerized applications:

docker build -t myapp:$(Build.BuildId) .
docker tag myapp:$(Build.BuildId) myregistry.azurecr.io/myapp:$(Build.BuildId)
docker push myregistry.azurecr.io/myapp:$(Build.BuildId)

Pros:

• Immutable container images
• Vulnerability scanning (Trivy integration)
• Long-term retention, geo-replication

Cons:

• Requires container runtime (Docker, Kubernetes)

Azure Blob Storage

For build artifacts (ZIP, tarball):

- task: AzureCLI@2
  inputs:
    azureSubscription: 'MyServiceConnection'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az storage blob upload \
        --account-name myartifacts \
        --container-name builds \
        --name backend-app-$(Build.BuildId).zip \
        --file dist.zip

Pros:

• Long-term storage, low cost
• Access control via SAS tokens or managed identity

Cons:

• Requires Azure Storage account setup

---

Environment-Specific Configuration

The Configuration Challenge

Your artifact is immutable, but each environment has different settings:

Setting	Dev	Test	Production
Database connection string	Dev SQL	Test SQL	Prod SQL
Frontend origin (CORS)	http://localhost:5173	https://test.example.com	https://app.example.com
Logging level	DEBUG	INFO	WARN

Solution: Externalize configuration — inject environment variables at deployment time, not build time.

Configuration Injection

- stage: DeployProd
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'

    - task: AzureWebApp@1
      inputs:
        appName: 'backend-prod'
        package: '$(System.DefaultWorkingDirectory)/backend-app'
        appSettings: |
          -NODE_ENV production
          -DATABASE_URL $(PROD_DATABASE_URL)
          -FRONTEND_ORIGIN https://app.example.com
          -KEY_VAULT_URL $(PROD_KEY_VAULT_URL)

Result: Same artifact deployed to all environments, configuration injected via Azure App Service settings or Key Vault references.

---

Approval Gates and Promotion Control

Manual Approval for Production

Even with the same artifact, you want human approval before production deployment:

- stage: DeployProd
  dependsOn: DeployTest
  condition: succeeded()
  pool:
    vmImage: 'ubuntu-latest'
  jobs:
    - deployment: DeployProduction
      displayName: 'Deploy to Production'
      environment: 'production'  # Requires manual approval in Azure DevOps
      strategy:
        runOnce:
          deploy:
            steps:
              - task: DownloadPipelineArtifact@2
                inputs:
                  artifactName: 'backend-app'
              # Deployment tasks...

In Azure DevOps:

1. Create an Environment named production
2. Add Approval and checks → Manual approval
3. Assign approvers (e.g., security team, operations lead)

Result: Pipeline pauses before production deployment, waits for approval, logs who approved and when.

---

Rollback Strategy: When Promoted Artifacts Fail

Artifact promotion is great for deploying forward, but what happens when a promoted artifact breaks production?

The Rollback Problem

Rollback Options

Strategy	How It Works	Pros	Cons	When to Use
Redeploy Previous Artifact	Download artifact from Build #455, deploy to production	Fast (minutes), tested artifact, no code changes	Doesn't fix the bug, only buys time	Immediate mitigation needed
Hotfix + New Artifact	Create hotfix branch, run full pipeline, promote new artifact	Fixes root cause, follows security gates	Slower (30-60 min), requires new build	Issue can be quickly identified and fixed
Feature Flag Rollback	Toggle feature off without redeploying	Instant (seconds), no deployment	Requires feature flags built into app	Feature-specific issue, flags already exist
Database Rollback	Revert database migrations	Restores data state	Risky (data loss), may not match code	Database migration caused the issue

Rollback Procedure

Step 1: Identify the Last Known Good Artifact

# Query Azure DevOps for recent successful deployments
az pipelines runs list --org https://dev.azure.com/myorg --project myproject \
  --branch main --status succeeded --top 10

Step 2: Download Previous Artifact

# Manual rollback pipeline
- task: DownloadPipelineArtifact@2
  inputs:
    buildType: 'specific'
    project: 'myproject'
    pipeline: 'backend-pipeline'
    buildVersionToDownload: 'specific'
    buildId: '455'  # Last known good build
    artifactName: 'backend-app'

Step 3: Deploy to Production

- task: AzureWebApp@1
  inputs:
    appName: 'app-course-prod'
    package: '$(System.DefaultWorkingDirectory)/backend-app'

Step 4: Verify Rollback

# Check deployed artifact metadata
curl https://app-course-prod.azurewebsites.net/api/version
# Expected: {"buildId": "455", "commit": "def789", "version": "1.2.2"}

Rollback Best Practices

Practice	Rationale
Retain artifacts for 90+ days	Allows rollback to any recent version
Tag rollback deployments	Clearly mark "ROLLBACK to Build #455" in deployment logs
Never skip testing after rollback	Even previous artifacts should be smoke-tested before production
Document rollback reason	Post-incident review: why did Build #456 fail?
Automate rollback pipeline	Pre-built "Deploy Specific Artifact" pipeline for emergencies

Rollback vs Hotfix Decision Tree

Production issue detected
  ↓
Is the cause known and fixable in <30 min?
  ├─ Yes → Hotfix + new artifact (#457)
  │         Run full pipeline: build → test → scan → deploy
  │
  └─ No → Rollback to previous artifact (#455)
            Buy time to investigate
            Then hotfix when root cause is identified

Azure DevOps Rollback Pipeline Example

# rollback-pipeline.yml
trigger: none  # Manual trigger only

parameters:
  - name: targetBuildId
    displayName: 'Build ID to rollback to'
    type: string
  - name: environment
    displayName: 'Environment'
    type: string
    default: 'production'
    values:
      - dev
      - test
      - production

stages:
  - stage: Rollback
    displayName: 'Rollback to Build ${{ parameters.targetBuildId }}'
    jobs:
      - deployment: RollbackDeployment
        environment: ${{ parameters.environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - task: DownloadPipelineArtifact@2
                  inputs:
                    buildType: 'specific'
                    buildId: '${{ parameters.targetBuildId }}'
                    artifactName: 'backend-app'
                
                - script: |
                    echo "ROLLBACK: Deploying artifact from Build #${{ parameters.targetBuildId }}"
                    cat backend-app/BUILD_COMMIT
                  displayName: 'Verify Rollback Artifact'
                
                - task: AzureWebApp@1
                  inputs:
                    appName: 'app-course-${{ parameters.environment }}'
                    package: 'backend-app'

Key Principle

Rollback is NOT a failure — it's a defensive measure. The ability to quickly revert to a known-good state is a security feature.

---

Immutable Artifact Validation

Checksum Verification

To prove the artifact hasn't been tampered with:

- stage: Build
  steps:
    - script: |
        cd dist
        sha256sum * > ../ARTIFACT_CHECKSUM.txt
      displayName: 'Generate Artifact Checksum'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: 'dist'
        artifactName: 'backend-app'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: 'ARTIFACT_CHECKSUM.txt'
        artifactName: 'backend-app-checksum'

- stage: DeployProd
  steps:
    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app'

    - task: DownloadPipelineArtifact@2
      inputs:
        artifactName: 'backend-app-checksum'

    - script: |
        cd $(System.DefaultWorkingDirectory)/backend-app
        sha256sum -c ../backend-app-checksum/ARTIFACT_CHECKSUM.txt
      displayName: 'Verify Artifact Integrity'

If checksums mismatch: Pipeline fails, preventing deployment of tampered artifact.

---

Key Questions Before Starting

Why is build-once-promote-many more secure than rebuild-per-environment?

Rebuilding introduces time drift (dependencies change), inconsistency (prod never tested), and untraceable artifacts (cannot prove prod matches scanned code). Build-once ensures the exact tested, scanned artifact is deployed everywhere.

How do you handle environment-specific configuration if the artifact is immutable?

Externalize configuration: Use environment variables, Azure App Service settings, or Key Vault references injected at deployment time. The artifact contains code, not configuration.

What if a hotfix is needed in production — do you promote from test?

No. Create a hotfix branch, run the full pipeline (build → test → scan → publish artifact), then promote the new artifact through dev → test → prod. Never skip environments or manually patch production.

Can you promote an artifact from last week's build to production?

Technically yes (if artifact is retained), but not recommended. Old artifacts may have unpatched dependencies or lack recent security fixes. Always promote recent builds that have passed all gates.

---

What You'll Do in Lab 3.6

• Review pipeline artifact publishing in Build stage (metadata files, published artifact)
• Download artifact in deployment stages (dev, test) and verify metadata
• Confirm no rebuilds occur in deployment stages (only DownloadPipelineArtifact)
• Validate traceability: Match deployed artifact to commit SHA, build ID, test results
• Configure approval gate for test environment (manual approval before deployment)
• Verify environment-specific configuration injection (no hardcoded values in artifact)

---

Before You Start

• Build stage publishes artifact with metadata (BUILD_VERSION, BUILD_COMMIT, BUILD_NUMBER)
• Deployment stages download artifacts, not rebuild
• Understand the difference between artifact storage (pipeline, ACR, Blob Storage)
• Know how to externalize configuration (environment variables, Key Vault)
• Azure DevOps environment resource created for approval gates

---

Lab File

Download Lab 3.6 — Artifact Promoting PDF