Artifact Management in CI/CD

Managing build artifacts is often an overlooked aspect of CI/CD pipelines, yet it’s fundamental to reliable, traceable, and efficient software delivery. Whether you’re shipping Docker images, npm packages, compiled binaries, or deployment bundles, how you store, version, and distribute these artifacts directly impacts your team’s velocity and system reliability. This guide explores enterprise-grade artifact management strategies used by companies scaling from dozens to thousands of deployments per day.

What Are Artifacts in CI/CD?

In the context of continuous integration and delivery, artifacts are the outputs of your build process that are used in subsequent stages of your pipeline or retained for auditing and rollback purposes.

Common Artifact Types:

Compiled Binaries

• Java JARs and WARs
• .NET assemblies and NuGet packages
• Go binaries
• Rust executables

Container Images

• Docker images
• OCI-compliant containers
• Helm charts

Package Manager Artifacts

• npm/yarn packages (tarballs)
• pip wheels and source distributions
• Maven artifacts (POM, JAR, sources)
• Ruby gems

Build Outputs

• Compiled frontend assets (bundled JS/CSS)
• Static site generators output
• Mobile app binaries (APK, IPA)
• Lambda deployment packages

Test and Documentation Artifacts

• Test reports (JUnit XML, coverage data)
• API documentation
• Build logs
• Security scan results

The key is treating these artifacts as immutable, versioned assets that flow through your deployment pipeline.

Why Artifact Management Matters

Traceability and Auditability

Every artifact should be traceable back to its source code commit, build number, and pipeline execution. When a production issue occurs, you need to answer: “What exact code is running in production?” Proper artifact management provides this visibility.

Netflix’s deployment system maintains complete artifact lineage. Every Docker image tag includes the git commit SHA, build timestamp, and CI job ID. This enables instant rollback to any previous version and precise debugging.

Reproducibility

Once built, an artifact should be deployable to any environment without rebuilding. Rebuilding for different environments introduces variables that can cause the infamous “works on my machine” problem at scale.

Spotify builds artifacts once in CI and promotes the same binary through dev, staging, and production. This ensures testing is performed on the exact code that reaches users.

Storage Optimization

Without proper management, artifact storage costs spiral. Google’s codebase produces terabytes of artifacts daily. Their artifact retention policies automatically delete old snapshots while preserving releases and critical versions.

Security and Compliance

Artifacts are software supply chain components. They must be scanned for vulnerabilities, signed for integrity, and stored securely. The SolarWinds attack highlighted how compromised build artifacts can become attack vectors.

Fundamentals of Artifact Management

Versioning Strategies

Choosing a versioning scheme is critical for artifact identification:

Semantic Versioning (SemVer)

1.2.3
major.minor.patch

Best for libraries and APIs with backward compatibility contracts.

Build Number + Git SHA

build.1234.abc123f

Provides direct traceability to source code. Used by companies like Uber.

Timestamp + Short SHA

2025.01.15.1430-abc123

Human-readable and sortable. Common in internal services.

GitFlow-Based Versioning

1.2.3-develop.45
1.2.3-rc.1
1.2.3

Reflects branching strategy. Useful for teams using GitFlow.

Example version strategy from Shopify:

# .gitlab-ci.yml
variables:
  VERSION: "${CI_COMMIT_REF_NAME}-${CI_PIPELINE_IID}-${CI_COMMIT_SHORT_SHA}"

build:
  script:
    - docker build -t myapp:${VERSION} .
    - docker tag myapp:${VERSION} myapp:latest

Storage and Retention Policies

Not all artifacts have equal value over time:

Critical Artifacts (Retain Indefinitely)

• Production releases
• Tagged versions
• Artifacts tied to regulatory requirements

Snapshot Artifacts (Short Retention)

• Development builds (7-30 days)
• Pull request builds (until PR closed)
• Failed builds (3-7 days)

Example Retention Policy:

# Artifactory cleanup policy
retention:
  releases:
    max_age: never  # Keep all releases
  snapshots:
    max_age: 30d
    max_count: 50
  pr_builds:
    max_age: 7d
    keep_latest: 10

Amazon S3 Lifecycle policies automate this:

{
  "Rules": [
    {
      "Id": "Delete old snapshots",
      "Filter": { "Prefix": "artifacts/snapshots/" },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    },
    {
      "Id": "Archive old releases",
      "Filter": { "Prefix": "artifacts/releases/" },
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Metadata and Tagging

Artifacts should carry metadata beyond just the version:

{
  "name": "myapp",
  "version": "1.2.3",
  "build_number": "4567",
  "git_commit": "abc123f",
  "git_branch": "main",
  "build_timestamp": "2025-01-15T14:30:00Z",
  "builder": "jenkins-agent-5",
  "test_status": "passed",
  "security_scan": {
    "vulnerabilities": "none",
    "scanner": "trivy",
    "scan_date": "2025-01-15T14:35:00Z"
  },
  "dependencies": {
    "nodejs": "18.17.0",
    "react": "18.2.0"
  }
}

Docker labels implement this:

LABEL org.opencontainers.image.created="2025-01-15T14:30:00Z" \
      org.opencontainers.image.revision="abc123f" \
      org.opencontainers.image.version="1.2.3" \
      build.number="4567" \
      build.pipeline="https://jenkins.example.com/job/4567"

Implementation Strategies

Artifact Repositories

Centralized artifact storage provides consistency and control:

Implementation Example: Artifactory in GitLab CI

variables:
  ARTIFACTORY_URL: "https://artifactory.company.com"
  DOCKER_REGISTRY: "${ARTIFACTORY_URL}/docker-local"

build:
  stage: build
  script:
    - mvn clean package
    - mvn deploy -DaltDeploymentRepository=artifactory::default::${ARTIFACTORY_URL}/libs-snapshot-local

docker_build:
  stage: package
  script:
    - docker build -t ${DOCKER_REGISTRY}/myapp:${CI_COMMIT_TAG} .
    - docker login -u $ARTIFACTORY_USER -p $ARTIFACTORY_PASSWORD ${ARTIFACTORY_URL}
    - docker push ${DOCKER_REGISTRY}/myapp:${CI_COMMIT_TAG}

Promoting Artifacts Through Environments

Build once, deploy many times. Artifact promotion ensures the same binary moves through environments:

# GitHub Actions promotion workflow
name: Promote to Production

on:
  workflow_dispatch:
    inputs:
      artifact_version:
        description: 'Version to promote'
        required: true

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - name: Pull artifact from staging
        run: |
          docker pull myregistry/myapp:${VERSION}-staging
          docker tag myregistry/myapp:${VERSION}-staging myregistry/myapp:${VERSION}-production
          docker push myregistry/myapp:${VERSION}-production

      - name: Update production deployment
        run: |
          kubectl set image deployment/myapp \
            myapp=myregistry/myapp:${VERSION}-production

Stripe’s deployment system tracks artifact promotions:

artifact: payment-service:1.2.3
  built: 2025-01-15 10:00 (commit: abc123)
  dev: deployed 2025-01-15 10:15 ✓
  staging: deployed 2025-01-15 14:00 ✓
  production: deployed 2025-01-16 09:00 ✓

Security and Scanning

Integrate security scanning into artifact lifecycle:

Container Image Scanning with Trivy:

security_scan:
  stage: test
  image: aquasec/trivy:latest
  script:
    - trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:${VERSION}
  artifacts:
    reports:
      container_scanning: trivy-report.json

Dependency Scanning:

dependency_check:
  stage: test
  script:
    - npm audit --audit-level=high
    - snyk test --severity-threshold=high

Artifact Signing:

Sign artifacts to verify integrity:

# Sign Docker image with Cosign
cosign sign --key cosign.key myregistry/myapp:1.2.3

# Verify signature before deployment
cosign verify --key cosign.pub myregistry/myapp:1.2.3

Google’s Binary Authorization requires signed container images before deploying to GKE clusters.

Advanced Techniques

Multi-Repository Strategy

Large organizations often maintain multiple artifact repositories:

Repository Types:

• Local repositories: Store internally built artifacts
• Remote repositories: Proxy external registries (npm, Docker Hub)
• Virtual repositories: Combine multiple repos behind single URL

Example Artifactory setup:

repositories:
  docker-local:
    type: local
    description: "Internal Docker images"

  docker-hub-remote:
    type: remote
    url: "https://registry-1.docker.io"
    description: "Proxy for Docker Hub"

  docker-virtual:
    type: virtual
    repositories:
      - docker-local
      - docker-hub-remote
    description: "Unified Docker registry"

Developers configure a single registry URL (docker-virtual) that pulls from local storage first, falling back to Docker Hub.

Artifact Deduplication

Identical artifacts waste storage. Content-addressable storage deduplicates:

Docker Layer Sharing:

# Base layer shared across images
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci

Multiple images sharing the node:18-alpine layer store it once.

Artifactory Storage:

Artifactory uses binary storage with checksum-based deduplication. Uploading identical artifacts multiple times stores them once.

Immutable Artifacts

Enforce immutability to prevent artifact tampering:

Docker Registry Configuration:

# Docker registry config.yml
storage:
  delete:
    enabled: false  # Prevent image deletion

validation:
  manifests:
    urls:
      allow:
        - ^https://trusted-registry\.com/

Artifactory Immutable Repos:

repositories:
  releases:
    type: local
    immutable: true  # Artifacts cannot be overwritten

Real-World Examples

Netflix’s Artifact Pipeline

Netflix builds artifacts in their CI system (Jenkins) and stores them in Artifactory. Key practices:

• Baked AMIs as artifacts: Pre-configured machine images stored and versioned
• Immutable deployments: AMIs never change after creation
• Retention: Critical releases kept indefinitely, snapshots for 90 days
• Metadata rich: Every AMI tagged with commit, build, tests, security scans

Etsy’s Artifact Strategy

Etsy deploys 50+ times daily with robust artifact management:

• One artifact per microservice: Each service produces a deployable artifact
• Artifact signing: All artifacts signed with GPG before deployment
• Promotion gates: Artifacts advance through environments after automated tests pass
• Audit trail: Complete history of what artifact deployed when and by whom

Airbnb’s Monorepo Artifacts

Airbnb’s monorepo produces hundreds of artifacts per build:

• Bazel for builds: Generates artifacts with content-based hashing
• Remote caching: Artifacts cached in Google Cloud Storage
• Selective deployment: Only changed services produce new artifacts
• Artifact graph: Dependency tracking shows which artifacts depend on which

Best Practices

DO

Tag Artifacts Consistently

# Good tagging scheme
myapp:1.2.3
myapp:1.2.3-abc123f
myapp:latest
myapp:stable

Automate Retention Cleanup

• Don’t rely on manual cleanup
• Use repository-level policies
• Monitor storage usage trends

Include Build Context

artifacts:
  name: "myapp-${CI_COMMIT_SHORT_SHA}"
  paths:
    - dist/
  reports:
    junit: test-results.xml
  metadata:
    git_commit: "${CI_COMMIT_SHA}"
    pipeline_url: "${CI_PIPELINE_URL}"

Separate Artifact Storage from Source

• Never commit built artifacts to git
• Use .gitignore aggressively
• Exception: vendored dependencies when necessary

Document Artifact Schema

Maintain artifact catalog:

# Artifact Catalog

## payment-service
- Type: Docker image
- Registry: registry.company.com/services
- Tagging: {version}-{shortSHA}
- Retention: Releases forever, snapshots 30d
- Size: ~450MB
- Dependencies: postgres:14, redis:7

DON’T

Avoid Mutable Tags

# Bad: Overwriting 'latest' loses history
docker build -t myapp:latest .
docker push myapp:latest

# Good: Unique tags for every build
docker build -t myapp:${VERSION} -t myapp:latest .
docker push myapp:${VERSION}
docker push myapp:latest

Don’t Store Secrets in Artifacts

• Use environment variables or secret management systems
• Scan for leaked credentials before publishing
• Implement pre-commit hooks to block secrets

Don’t Skip Vulnerability Scanning

Every artifact should pass security scans:

- trivy image --severity CRITICAL --exit-code 1 $IMAGE

Avoid Over-Retention

Storing everything forever is expensive and noisy. Set sensible defaults:

• Snapshots: 7-30 days
• PRs: Until merged/closed + 7 days
• Releases: Indefinite or compliance-driven

Common Pitfalls

Inconsistent Versioning

Problem: Different teams using different versioning schemes causes confusion.

Solution: Enforce repository-wide versioning standard:

# .version-policy.yml
strategy: semver
prefix: "v"
snapshot_suffix: "-SNAPSHOT"

Storage Sprawl

Problem: Artifacts stored in multiple locations (S3, Artifactory, local servers).

Solution: Consolidate to centralized repository with proper access controls.

Missing Metadata

Problem: Unable to trace artifact back to source or build.

Solution: Always include:

• Git commit SHA
• Build number
• Build timestamp
• Pipeline URL

No Rollback Plan

Problem: Previous artifacts deleted, making rollback impossible.

Solution: Retention policy must keep N previous production versions (minimum 3).

Tools and Resources

Artifact Repository Solutions

Security Tools

• Trivy: Container vulnerability scanner
• Snyk: Dependency vulnerability detection
• Cosign: Container image signing
• Notary: Docker content trust
• OWASP Dependency-Check: Dependency vulnerability scanner

Monitoring and Analytics

• Artifactory Query Language (AQL): Search and analyze artifact metadata
• Datadog Artifacts Integration: Monitor artifact usage and storage
• Grafana + Prometheus: Track registry metrics

Official Documentation

• JFrog Artifactory Docs
• AWS ECR User Guide
• Google Artifact Registry

Measuring Success

Track these metrics for effective artifact management:

Storage Efficiency

Storage Efficiency = (Deduplicated Size / Raw Size) × 100

Artifact Age Distribution

• Percentage of artifacts < 30 days old
• Percentage > 90 days (candidates for cleanup)

Download/Upload Ratio

Ratio = Downloads / Uploads

High ratio indicates artifacts are reused effectively.

Time to Promote

Promotion Time = Production Deployment Time - Artifact Build Time

Track to identify promotion bottlenecks.

Conclusion

Effective artifact management is the backbone of reliable CI/CD. By treating artifacts as first-class citizens with proper versioning, metadata, security, and lifecycle management, you enable:

• Faster deployments: Promote artifacts instantly without rebuilds
• Better traceability: Know exactly what code is running where
• Reduced costs: Optimize storage with retention policies
• Enhanced security: Scan and sign artifacts consistently

Start with a centralized artifact repository, implement consistent versioning, and automate retention policies. Build on this foundation with security scanning, artifact signing, and promotion workflows.

The examples from Netflix, Etsy, and Airbnb demonstrate that even at massive scale, proper artifact management keeps deployments fast, safe, and auditable.

Next Steps:

1. Choose an artifact repository that fits your stack
2. Define and document your versioning strategy
3. Implement automated retention policies
4. Add security scanning to your build pipeline
5. Set up artifact promotion workflows between environments

For more CI/CD best practices, explore our guides on caching strategies and pipeline optimization.