What Is a Service Mesh?

A service mesh is an infrastructure layer that manages communication between microservices. Instead of each service implementing its own retry logic, circuit breakers, and encryption, these concerns are handled by a proxy (sidecar) that sits alongside each service.

The most common service mesh implementations are:

Service MeshProxyKey Features
IstioEnvoyFull-featured, widely adopted, complex
Linkerdlinkerd2-proxyLightweight, Rust-based, simpler
Consul ConnectEnvoyHashiCorp ecosystem integration

How It Works 

┌──────────────────┐     ┌──────────────────┐
│  Service A        │     │  Service B        │
│  ┌──────────────┐│     │┌──────────────┐  │
│  │  App Code    ││     ││  App Code    │  │
│  └──────┬───────┘│     │└──────▲───────┘  │
│         │        │     │       │          │
│  ┌──────▼───────┐│     │┌──────┴───────┐  │
│  │  Sidecar     │├────▶││  Sidecar     │  │
│  │  Proxy       ││     ││  Proxy       │  │
│  └──────────────┘│     │└──────────────┘  │
└──────────────────┘     └──────────────────┘

All traffic between services flows through the sidecar proxies. The mesh control plane configures these proxies with routing rules, security policies, and observability settings.

What to Test in a Service Mesh

1. Traffic Routing

Service meshes allow sophisticated traffic routing: canary deployments, A/B testing, header-based routing, and traffic splitting.

Canary deployment test:

# Istio VirtualService: 90% to v1, 10% to v2
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: product-service
spec:
  hosts:
    - product-service
  http:
    - route:
        - destination:
            host: product-service
            subset: v1
          weight: 90
        - destination:
            host: product-service
            subset: v2
          weight: 10

How to verify: Send 1,000 requests and check that roughly 900 go to v1 and 100 go to v2. Use the mesh’s telemetry data to verify traffic distribution.

2. Load Balancing

Verify the mesh distributes traffic across service instances using the configured algorithm (round-robin, least connections, random).

Test approach:

  1. Deploy 3 replicas of a service.
  2. Send 100 requests.
  3. Check access logs or metrics to verify each replica received roughly equal traffic.

3. Circuit Breakers

Circuit breakers prevent cascading failures by stopping requests to an unhealthy service.

# Istio DestinationRule with circuit breaker
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: product-service
spec:
  host: product-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 60s
      maxEjectionPercent: 50

Test approach:

  1. Make one instance of the service return 500 errors.
  2. Send traffic and verify the mesh ejects the failing instance after 5 consecutive 5xx errors.
  3. Verify traffic reroutes to healthy instances.
  4. Fix the failing instance and verify it re-enters the pool after the ejection period.

4. Fault Injection

The mesh can inject faults without changing application code — invaluable for resilience testing.

Inject a 5-second delay:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: product-service
spec:
  hosts:
    - product-service
  http:
    - fault:
        delay:
          percentage:
            value: 100
          fixedDelay: 5s
      route:
        - destination:
            host: product-service

Inject HTTP 503 errors:

http:
  - fault:
      abort:
        percentage:
          value: 50
        httpStatus: 503
    route:
      - destination:
          host: product-service

Test approach: Apply the fault injection, then verify that calling services handle the fault gracefully (retry, fallback, circuit break).

5. mTLS Verification

Verify that mutual TLS is enforced between services.

Test approach:

  1. Check the mesh configuration to confirm mTLS is in STRICT mode (not PERMISSIVE).
  2. Try to call a service directly (bypassing the sidecar proxy) — it should be rejected.
  3. Use istioctl or mesh dashboards to verify all traffic is encrypted.
# Istio: Check mTLS status
istioctl x describe pod <pod-name>

# Verify STRICT mode
kubectl get peerauthentication -A

6. Observability

Service meshes provide metrics, distributed traces, and access logs automatically.

Verify:

  • Prometheus metrics are collected for every service-to-service call.
  • Distributed traces (Jaeger/Zipkin) show the full request path.
  • Access logs capture request/response details.

Testing Service Mesh in Practice

Local Testing with Minikube 

# Start Minikube with enough resources for Istio
minikube start --cpus=4 --memory=8192

# Install Istio
istioctl install --set profile=demo

# Enable sidecar injection for your namespace
kubectl label namespace default istio-injection=enabled

# Deploy your services
kubectl apply -f k8s/

Verifying Mesh Behavior

Use istioctl proxy-status to verify all sidecars are synced with the control plane:

istioctl proxy-status
# All services should show SYNCED status

Use istioctl analyze to detect configuration issues:

istioctl analyze
# Reports misconfigurations, unused resources, and conflicts

Exercise: Service Mesh Testing Scenarios

Set up a service mesh environment and test its key features.

Setup

If you have a Kubernetes cluster with Istio installed, use your own services. Otherwise, use Istio’s Bookinfo sample application:

# Deploy Bookinfo sample
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/platform/kube/bookinfo.yaml

# Create gateway
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/bookinfo/networking/bookinfo-gateway.yaml

# Verify deployment
kubectl get pods

Task 1: Traffic Splitting

Configure a canary deployment for the reviews service:

  1. Apply a VirtualService that routes 80% of traffic to reviews-v1 and 20% to reviews-v3.
  2. Send 50 requests and record which version each request reached.
  3. Verify the traffic split is approximately 80/20.
# Send requests and count versions
for i in $(seq 1 50); do
  curl -s http://$GATEWAY_URL/productpage | grep -o 'reviews-v[0-9]' >> results.txt
done
sort results.txt | uniq -c

Task 2: Fault Injection

Inject a 7-second delay into the ratings service:

  1. Apply a fault injection VirtualService.
  2. Access the productpage and observe the behavior.
  3. Does the page load? Does it show an error? How long does it take?
  4. Document how the upstream services (reviews, productpage) handle the delay.

Task 3: Circuit Breaker

Configure a circuit breaker for the ratings service:

  1. Apply a DestinationRule with outlier detection (3 consecutive 5xx errors, 30-second ejection).
  2. Make the ratings service return errors (scale it to 0 replicas or inject 503 faults).
  3. Send traffic and observe when the circuit breaker trips.
  4. Restore the ratings service and verify it re-enters the pool.

Task 4: mTLS Audit

Verify encryption between services:

  1. Check the PeerAuthentication policy — is mTLS STRICT or PERMISSIVE?
  2. Use Kiali or istioctl to verify all connections use mTLS.
  3. If PERMISSIVE, try sending an unencrypted request between services and document whether it succeeds.

Deliverables

  1. A test report documenting the results of each task.
  2. Screenshots or command output showing traffic distribution, fault behavior, circuit breaker activation, and mTLS status.
  3. A list of configuration issues found (if any) and recommendations.