Deployment Strategies — Complete Engineering Reference

Q01What is the difference between Blue-Green and Canary deployments? When would you choose one over the other?Easy

Blue-Green maintains two full identical environments and switches all traffic at once — optimized for instant rollback and zero downtime. Canary routes a small percentage of traffic to the new version progressively.

Choose Blue-Green when: rollback speed is critical (payments, auth), you have budget for 2× infra, and your release is binary (all-or-nothing). Choose Canary when: you want to validate with real traffic before full exposure, you have good observability, and can tolerate a small blast radius during validation.

Q02How do you handle database migrations in a Blue-Green deployment?Hard

The hardest part of Blue-Green. Both Blue (v1) and Green (v2) will share the same database simultaneously during the switch window. Use the Expand-Contract pattern (also called Parallel Change):

Expand phase: Add new columns/tables. Old code (Blue) ignores new columns. New code (Green) uses both old and new.
Migrate phase: Traffic is live on Green. Backfill data from old structure to new.
Contract phase: Once Blue is decommissioned, drop old columns/tables in the next release cycle.

Never run destructive migrations (DROP COLUMN, RENAME) during a Blue-Green deployment window.

Q03What is a Rolling deployment and what are maxSurge and maxUnavailable?Easy

A Rolling deployment replaces instances of the old version incrementally with the new version, ensuring some capacity is always available. In Kubernetes:

maxSurge: Maximum number of extra pods above the desired replica count during a rollout. E.g., maxSurge: 1 with 4 replicas allows 5 pods temporarily.
maxUnavailable: Maximum number of pods that can be unavailable during the update. E.g., maxUnavailable: 1 means at most 1 pod can be down.

Setting both to 0 is invalid. Typical production config: maxSurge: 25%, maxUnavailable: 25%.

Q04Explain Shadow deployment. What are the risks and how do you isolate side effects?Hard

Shadow deployment mirrors real production traffic to a new version asynchronously. The shadow response is discarded — users never see it.

Key risks:

Write side effects: Shadow requests that hit the same DB will cause double-writes. Solution: use a separate shadow DB, or implement a shadow-only write path with a x-shadow: true header that services intercept.
Third-party calls: Shadow requests may trigger real emails, payments, or API calls. Solution: stub external calls in shadow mode using if (req.headers['x-shadow']) return mockResponse().
Infrastructure load: Doubles compute. Set async timeouts so shadow failures never impact the live path.

Q05How do you handle user sessions and authentication during a Blue-Green switch?Medium

Sessions can break if they're stored in-memory or in a local store on Blue servers, because after the switch, Green servers have no access to them.

Solutions:

JWT (preferred): Stateless tokens — no server-side session state. Green can validate tokens without any shared store.
Redis session store: Both Blue and Green connect to the same Redis cluster. Sessions survive the switch.
Sticky sessions: The load balancer pins a user to the same server (Blue or Green) during the transition window. Adds complexity and defeats some rollback benefits.

Best practice: design services as stateless. JWT + Redis for anything that must be stateful.

Q06What metrics do you monitor during a Canary release? What triggers an automatic rollback?Medium

Primary metrics to watch:

Error rate: HTTP 5xx rate. Trigger if > 0.1% above baseline
Latency: p50, p95, p99 — alert if p99 increases >10%
Saturation: CPU, memory, DB connection pool usage
Business metrics: conversion rate, checkout success, login success rate

Automatic rollback triggers: Use tools like Flagger (Kubernetes), AWS CodeDeploy hooks, or custom health-check scripts. A common pattern: if error rate > 1% over a 5-minute window, automatically reroute all traffic back to stable and page on-call. Implement with Prometheus + AlertManager + a webhook that triggers a deployment rollback.

Q07What is the difference between A/B testing deployment and Canary deployment?Easy

Canary is a deployment safety mechanism: "Is this version stable enough to fully replace v1?" Traffic is split randomly, the goal is to detect regressions early, and it ends when v2 reaches 100%.

A/B Testing is a product experimentation mechanism: "Does this version produce better business outcomes for specific users?" Traffic is split by user attributes (userId, plan, region), the goal is to measure conversion/engagement, and it may run for weeks or indefinitely as a feature flag.

They can run simultaneously: a canary rollout of a feature that is also being A/B tested by user segment.

Q08You are deploying a Node.js microservice that calls a third-party payment API. How would you approach the deployment strategy?Hard

This is a scenario question testing architectural judgment. Use Blue-Green with careful considerations:

Idempotency keys: Ensure payment requests have idempotency keys so that if a request is processed by both Blue and Green during a brief overlap, the payment provider deduplicates it.
Circuit breaker: Both Blue and Green should have independent circuit breakers to the payment API. Use a library like opossum in Node.js.
Request tracing: Add a x-deployment-version header to outgoing payment calls so you can correlate issues in payment provider logs with your deployment version.
Canary first, then Blue-Green: Route 1% of traffic to Green → verify zero payment failures → flip fully to Green.

Q09How does Kubernetes implement Rolling updates internally? What happens if a pod's readiness probe fails?Medium

Kubernetes rolling update process:

1. Creates a new ReplicaSet for v2 while keeping v1 ReplicaSet alive
2. Scales up v2 pods one by one (respecting maxSurge)
3. For each new pod, waits for readiness probe to pass
4. Only then scales down one v1 pod (respecting maxUnavailable)

If readiness probe fails: The new pod never receives traffic. The rollout halts. Old pods continue serving. The deployment enters a degraded/stalled state. You can set progressDeadlineSeconds (e.g., 600s) after which Kubernetes marks the deployment as failed and you can configure automated rollback via kubectl rollout undo or CI/CD hooks.

Q10What is the "Expand-Contract" (Parallel Change) pattern and when is it required?Hard

Expand-Contract is a DB migration pattern required when old and new versions of code must coexist with the same database (Rolling, Blue-Green, Canary).

Phases:

Expand: Add new columns/tables in a backward-compatible way. Both v1 (ignores new) and v2 (uses new) can run simultaneously.
Migrate: Backfill data. Both versions are compatible. The transition completes.
Contract: In the next release (after v1 is fully gone), remove the old columns/tables.

Required any time you need to: rename a column, change a column type, split a table, or merge tables. The contract phase is often forgotten and leads to DB bloat.

Q11How would you implement Blue-Green deployment in AWS without Kubernetes?Medium

AWS-native Blue-Green options:

AWS CodeDeploy: Native Blue-Green support for EC2 Auto Scaling Groups and Lambda. Automatically shifts traffic, runs health checks, and rolls back. Supports AllAtOnce, HalfAtATime, OneAtATime traffic shifts.
ALB Weighted Target Groups: Create two Target Groups (Blue TG, Green TG) behind an Application Load Balancer. Use aws elbv2 modify-rule to adjust weights from 100/0 to 0/100.
ECS with CodeDeploy: AWS ECS + CodeDeploy integration provides built-in Blue-Green with automatic task set switching.
Lambda Traffic Shifting: Lambda Aliases support weighted traffic splitting natively — perfect for serverless canary/blue-green.

Q12Design a deployment strategy for a high-traffic Node.js e-commerce checkout API (50k RPM). Walk me through your full approach.Hard

This is a full system design answer. Recommended approach: Canary → Blue-Green hybrid

Infrastructure: Kubernetes on EKS, 3 AZs, ALB with weighted target groups
Phase 1 – Canary (1%): Deploy v2 pods, route 1% of checkout traffic. Monitor: error rate, p99 latency, payment success rate, cart abandonment rate. Duration: 30 minutes.
Phase 2 – Expand (5% → 25% → 50%): Automated promotion if error rate < 0.05% for 10 consecutive minutes per stage.
Phase 3 – Blue-Green flip: At 50% validation, do a full flip to 100% v2. Blue remains warm for 30 minutes.
Database: Expand-Contract pattern, separate read replica for v2 testing. Redis sessions shared.
Rollback SLA: <60 seconds via automated Flagger rollback + PagerDuty alert.

Q13What is a feature flag and how does it differ from a deployment strategy?Easy

Feature flags (feature toggles) decouple code deployment from feature activation. You deploy code with the feature disabled, then turn it on in production without a new deployment.

Deployment strategy: Controls how the binary/container is replaced on servers (Blue-Green, Rolling, etc.)
Feature flag: Controls which users see which features at runtime, independent of deployment.

They complement each other: you can do a Blue-Green deployment of code that has a feature flag — all instances (Blue and Green) get the new code, but the feature is disabled until you flip the flag in LaunchDarkly or your own flag service. This gives you "dark launches" — ship code safely, activate features on your schedule.

Q14How do you prevent cache poisoning during a Blue-Green switch when both environments share a Redis cache?Hard

Cache poisoning happens when v2 writes a new data format to Redis that v1 can't deserialize (or vice versa during a rollback).

Strategies:

Version-namespaced cache keys: Prefix all keys with the deployment version — v1:user:123 vs v2:user:123. Both versions use isolated cache namespaces. Cache warm-up cost on switch.
Backward-compatible serialization: If changing cache value schema, make v2 able to deserialize both v1 and v2 formats. v2 writes new format, v1 writes old.
Cache flush on switch: Acceptable for non-critical caches. Add a warm-up phase (send synthetic traffic to pre-warm Green before switching).
Blue-Green Redis instances: Separate Redis per environment. More expensive but eliminates all cache conflict risks.

Q15What is a "readiness probe" vs "liveness probe" in Kubernetes and why are both critical for zero-downtime deployments?Medium

Liveness Probe: "Is this container still alive?" If it fails, Kubernetes restarts the container. Detects deadlocks, OOM, corrupted state. Node.js example: /health/live returns 200 if the event loop is responsive.

Readiness Probe: "Is this container ready to receive traffic?" If it fails, Kubernetes removes the pod from the Service endpoints (stops routing traffic to it). Does NOT restart the container. Node.js example: /health/ready checks DB connection, Redis connection, and any warm-up completion.

Why both matter for zero-downtime: During a Rolling update, a new pod is only added to the load balancer pool when its readiness probe passes. This ensures no traffic is sent to a pod that hasn't finished initializing (DB pool warming, cache loading, etc.). Without readiness probes, you get 502 errors on deployment.

Q16How do you handle in-flight requests during a Blue-Green switch to avoid dropped connections?Hard

When flipping from Blue to Green, in-flight requests on Blue must complete before Blue shuts down.

Solution: Graceful shutdown in Node.js

Listen for SIGTERM signal (sent by K8s/Docker on shutdown)
Stop accepting new connections: server.close()
Wait for existing connections to drain (track active requests with a counter)
Set a hard timeout (30s) in case connections never close
In Kubernetes, set terminationGracePeriodSeconds: 30 and add a preStop: sleep 5 hook to account for iptables propagation delay before Blue stops getting traffic

At the load balancer level, configure connection draining (AWS ALB: "Deregistration delay" = 30s) so the LB waits for in-flight requests to complete before removing Blue from the target group.

Q17When would you NOT use Canary deployment despite its benefits?Medium

Canary is not always the right choice. Avoid it when:

Breaking API changes: If v2's API is incompatible with v1, routing some users to v2 and others to v1 will break cross-service calls. Use Blue-Green instead.
Stateful workloads: Databases, message queues, or services with sticky state that can't run two versions simultaneously.
Regulatory compliance: Some industries (finance, healthcare) require all users to be on the same version simultaneously for audit consistency.
Poor observability: Canary without good metrics and alerting is worse than no canary — you might not detect problems until they affect 100% of traffic.
Low-traffic services: With <100 RPM, 5% canary traffic is too small to produce statistically meaningful error signals.

Q18Explain how Flagger automates Canary deployments on Kubernetes.Hard

Flagger is a Kubernetes operator that automates the Canary release process using a custom resource called Canary.

How it works:

You define a Canary CRD with analysis thresholds (error rate, latency) and traffic weight steps (5% → 10% → 25% → 50% → 100%)
Flagger creates a primary deployment (stable) and a canary deployment (new) automatically
It queries Prometheus/Datadog metrics to evaluate health after each step
If metrics pass: automatically advances to next weight increment
If metrics fail: automatically rolls back to 0% canary and alerts
Supports Istio, Linkerd, NGINX, and AWS ALB for traffic shifting

Flagger eliminates the manual promotion/rollback steps that are error-prone in canary deployments.

Q19You deployed a new version and users report bugs. Walk me through your complete rollback procedure for each strategy.Medium

Blue-Green: Update load balancer config to point back to Blue. Run nginx -s reload or update ALB target group weights. Time: ~5 seconds. Zero impact.
Canary: Set canary weight to 0% in your traffic controller (Flagger, NGINX, Istio). All traffic returns to stable v1. Time: ~10 seconds.
Rolling (K8s): Run kubectl rollout undo deployment/name. Kubernetes starts a new rolling update back to the previous version. Time: minutes (depends on replica count).
Shadow: No rollback needed — users were never exposed to the shadow version. Simply stop mirroring traffic.
Recreate: Re-deploy the old version. Has downtime. This is why Recreate is risky in production.
A/B Testing: Update routing rules to send all segments to the stable version. Feature flag off.

Q20How do microservice dependencies affect deployment strategy choice?Hard

Microservices add complexity because Service A's deployment may break Service B if their contracts change.

Key principles:

Consumer-driven contracts (Pact): Service consumers define what they expect. Producers must satisfy contracts before deploying. Prevents breaking changes from reaching production.
Backward compatibility window: When changing an API, maintain the old endpoint for at least one full deployment cycle. Rolling or Canary deployments may have the caller on v1 while the callee is on v2.
Deploy dependencies first: If Service B depends on Service A, deploy A first (with backward-compatible additions), then deploy B, then clean up A's old endpoints.
Prefer Canary over Rolling for breaking changes: Canary lets you validate the full service graph before committing. Rolling exposes all instances to the change gradually.
Service mesh (Istio/Linkerd): Enables fine-grained traffic control per service — run canary and rolling strategies independently across your service mesh.

Q21What is a "dark launch" and how does it relate to Shadow deployment?Easy

A dark launch means deploying code to production but keeping the feature invisible to users. The code is live but inactive.

Relationship to Shadow: They're related but different. Shadow deployment mirrors traffic to test new infrastructure. A dark launch deploys code that's controlled by feature flags.

Example: You dark-launch a new recommendation engine by deploying it with a feature flag set to false. Then you enable Shadow mode — the engine processes real traffic in parallel, and you compare its recommendations to the current engine's. Once confident, you enable the flag for 1% of users (canary). This is the full progression: dark launch → shadow → canary → full release.

Q22How do you implement zero-downtime deployment for a WebSocket-heavy Node.js service?Hard

WebSocket connections are long-lived — dropping them during a deploy causes user-visible disruptions (disconnects, lost messages).

Strategy: Blue-Green with graceful WebSocket drain

Client-side reconnect logic: Implement exponential backoff reconnection on the frontend. This is mandatory — any server restart will disconnect WebSockets.
Graceful shutdown: On SIGTERM, stop accepting new WS connections, send a { type: "server_restart", reconnect_in: 3000 } message to all connected clients, then wait for clients to acknowledge and reconnect to Green.
Session handoff via Redis: Store active subscription state in Redis. Green picks up the subscription state when the client reconnects.
Sticky sessions at LB: During the switch, pin existing WebSocket sessions to Blue via LB session affinity. New connections go to Green. Old connections drain naturally.
No Rolling for WebSocket services: Rolling causes partial reconnects. Use Blue-Green where the switch is clean and the drain window is explicit.

Q23Compare CI/CD pipeline design for Blue-Green vs Rolling deployments.Medium

Rolling CI/CD pipeline (simpler):
Build → Test → Push image → kubectl set image → Monitor rollout → Auto-rollback on probe failure

Blue-Green CI/CD pipeline (more complex):
Build → Test → Push image → Deploy to inactive environment (Green) → Run smoke tests against Green → Validate health checks → Switch LB traffic → Monitor → Decommission Blue (after TTL) OR keep warm for rollback

Key difference: Blue-Green requires a validation gate between deploy and traffic switch. Rolling is continuous. Tools: GitHub Actions + Helm for Rolling; AWS CodeDeploy or custom scripts with ALB API for Blue-Green. For Kubernetes Blue-Green, tools like Argo Rollouts provide native support without manual scripting.

Q24What is "traffic shadowing" in Istio and how does it compare to application-level mirroring?Hard

Istio traffic mirroring (also called shadowing) mirrors a percentage of live traffic to a shadow service at the infrastructure level — no application code changes needed.

Istio VirtualService config example: mirror: host: v2-shadow with mirrorPercentage: 10 — sends 10% of requests to shadow, 100% to production. The shadow response is dropped by Istio.

vs. Application-level mirroring (custom middleware):

Istio: No code changes, infrastructure-managed, percentage-based, cleaner isolation
App-level: More flexible (can modify shadow request, log diffs inline), works without a service mesh, but adds latency risk to production path if not properly async

For production at scale: Istio/Envoy mirroring is preferred. For quick validation or ML testing: app-level middleware is faster to implement.

Q25You're a tech lead. A junior dev asks: "Why not just always use Blue-Green? It seems safest." How do you respond?Medium

Great question — Blue-Green is excellent, but it's not universally optimal:

Cost: 2× infrastructure at all times. For a 200-service microservice platform, that's a 2× cloud bill permanently. At scale, Canary or Rolling is more economical.
Database complexity: Blue-Green's hardest problem is DB compatibility. Rolling/Canary with Expand-Contract migrations are often easier to manage in practice.
Stateful services: Blue-Green assumes stateless services. Databases, Kafka consumers, and job workers can't simply be "doubled" safely.
Warm-up time: Green needs to be warm (JVM JIT, Node.js V8 caches, connection pools). Under load, cold-start performance can hurt users right after the switch.
Not always necessary: For dev/staging — Recreate is perfectly fine. For an internal tool with 10 users — Rolling is more than enough.

The answer is always: match the strategy to the service's criticality, traffic, and team's operational maturity.

🎯 25 Interview Questions

Deployment Interview Questions