AWS Builder Center
Load Balancing in Distributed Systems

Load Balancing in Distributed Systems

Learn how load balancing distributes traffic, prevents server overload, and improves the reliability of distributed applications. This practical guide explains server load, the OSI model, Layer 4 and Layer 7 routing, static and dynamic balancing algorithms, and AWS Network and Application Load Balancers. You will also build and test your own load balancer using FastAPI, Nginx, Docker Swarm, and k6, with private container networking, health checks, timeouts, scaling, monitoring, and performance thresholds.

Series: Distributed Systems and Architecture (1 article)

  1. 1
    Load Balancing in Distributed Systems This article
Modern applications fail when requests overwhelm finite resources, servers become unhealthy, or traffic is distributed poorly. Load balancing addresses these problems by placing a traffic-distribution layer between clients and application instances.
Today, we will learn how load balancers work, explore common load-balancing strategies, implement load balancing on AWS, and build a load balancer from scratch.

What is Load Balancing?

Load balancing is the process of accepting work at a shared entry point and distributing it among multiple eligible workers.
A worker performs the forwarded work and may be a server, container, virtual machine, function, process, or private IP endpoint. It is eligible only while reachable and healthy. Capacity differences should be reflected in the balancing strategy. For web applications, the entry point is typically a DNS name and virtual IP.
A load balancer typically performs four jobs:
  • Accept traffic on one or more listeners, such as TCP port 443 or HTTPS port 443.
  • Select a backend from a pool according to an algorithm.
  • Avoid backends that fail health checks.
  • Forward the connection or request and, depending on the product, return the response through the same path.
Load balancing improves scalability and availability, but it is not the same thing as scaling. A load balancer can distribute traffic only across the capacity that already exists. An autoscaler adds or removes capacity; the load balancer then discovers the updated pool and routes traffic to its healthy members.

What is load?

Load is the demand placed on a system during a period of time. Requests per second is a useful measure, but not a complete one. A system may also be loaded by:
  • concurrent TCP connections;
  • active HTTP requests;
  • bytes per second;
  • CPU time and run-queue length;
  • allocated memory and garbage-collection pressure;
  • disk or network I/O;
  • database connections, locks, or queue depth;
  • external API calls; and
  • accelerators or other scarce resources.
One request can affect several resources: TLS and JSON consume CPU; buffers and objects consume memory; database calls occupy connections; and uploads use network and disk I/O. A slow dependency keeps state alive longer, increasing concurrency even when request rate is unchanged.
This relationship can be approximated with Little's Law:
 average requests in flight ≈ arrival rate × average time in the system 
At 500 requests per second and an average response time of 200 milliseconds, roughly 100 requests are in flight. If a dependency slows the average to 2 seconds, the same arrival rate results in roughly 1,000 in-flight requests. Memory use, open sockets, and queueing can therefore increase tenfold without any increase in requests per second.

Why One Server Has a Limit?

A server has a finite number of CPU cores, bytes of memory, sockets, file descriptors, worker threads, event-loop capacity, and downstream connections. Once a bottleneck reaches saturation, additional work waits in a queue. Queueing raises latency; client retries can add more traffic; timeouts and failures follow. Adding backends and balancing traffic across them increases aggregate capacity and allows traffic to continue when one backend is removed or fails.
Good load distribution is therefore not necessarily the same as request counts. If one request takes 10 milliseconds and another takes 10 seconds, sending one of each to two servers is numerically even but operationally uneven. The choice of algorithm determines which signals the balancer uses and which signals it ignores.

OSI model of developing Applications

The OSI model separates network communication into seven conlceptual layers; it is not a seven-tier application architecture. See AWS's  OSI overview .
Implementations do not always fit neatly into one layer. For load balancers, the layer usually describes the information used for routing decisions, not every feature.

Layer 7: Application

  • Responsibility: Provides network services directly to user-facing applications and defines the meaning and structure of application messages.
  • What is present: URLs, HTTP methods, headers, cookies, content types, status codes, API operations, message formats, and application-aware authentication.
  • Examples: HTTP/1.1, HTTP/2, HTTP/3, WebSocket, gRPC, DNS, and SMTP.
  • Application-development view: Developers define routes such as GET /orders/{id}, validation, request and response bodies, authorization, and error contracts. A Layer 7 proxy can inspect these details and route /images/* and /api/* to different target groups.

Layer 6: Presentation

  • Responsibility: Translates data so both ends agree on encoding, serialization, compression, and encryption.
  • What is present: Character encodings, serialized values, compressed payloads, encryption state, and negotiated data formats.
  • Examples: UTF-8, JSON, Protocol Buffers, JPEG, gzip, and the representation aspects of TLS.
  • Application-development view: A client may send JSON compressed with gzip over an encrypted connection. An application or proxy decodes those representations before business logic can use the data. In the Internet protocol suite, these concerns are commonly implemented inside libraries and application protocols rather than as a distinct network service.

Layer 5: Session

  • Responsibility: Establishes, maintains, synchronizes, and closes a logical conversation between applications.
  • What is present: Session identifiers, dialog state, checkpoints, reconnection state, and keepalive behavior.
  • Examples: RPC sessions, database sessions, session tokens, and protocol-level dialog management. Applications often use cookies to recreate session behavior because HTTP is stateless.
  • Application-development view: Stateful applications may use session affinity, or sticky sessions. Externalizing session state to a database or cache lets any healthy backend serve the next request.

Layer 4: Transport

  • Responsibility: Provides end-to-end transport between processes, including ports, segmentation, reliability, ordering, congestion handling, or connectionless delivery.
  • What is present: Source and destination ports, TCP connection state, sequence and acknowledgment numbers, and UDP datagrams.
  • Examples: TCP and UDP. QUIC runs over UDP and provides transport features in user space.
  • Application-development view: A process listens on a port such as TCP 8000. TCP provides a reliable ordered stream; UDP provides lower-overhead datagrams. A Layer 4 balancer routes using addresses, ports, and protocol without interpreting the application bytes.

Layer 3: Network

  • Responsibility: Moves packets between networks and selects paths through routers.
  • What is present: Source and destination IP addresses, routing tables, hop limits or TTL, and fragmentation information.
  • Examples: IPv4, IPv6, ICMP, routers, VPC route tables, and IP subnets.
  • Application-development view: This layer determines client-to-balancer and balancer-to-target reachability. VPC CIDRs, subnet routes, VPNs, and transit routing live primarily here.

Layer 2: Data Link

  • Responsibility: Transfers frames across the same local link and detects some link-level errors.
  • What is present: Frames, MAC addresses, VLAN tags, and link-level checksums.
  • Examples: Ethernet, Wi-Fi MAC, VLANs, network interface cards, and switches.
  • Application-development view: Developers usually interact with this layer indirectly, but MTU, VLAN, and neighbor-discovery faults can appear as intermittent application failures.

Layer 1: Physical

  • Responsibility: Transmits raw bits as electrical, optical, or radio signals.
  • What is present: Voltages, frequencies, light pulses, modulation, connectors, cabling, and physical link rates.
  • Examples: Copper Ethernet, fiber, Wi-Fi radio, transceivers, and repeaters.
  • Application-development view: Cloud providers abstract this layer, but physical loss and capacity still constrain every higher layer.

Layer 4 and Layer 7 Load Balancers

An HTTPS request still uses IP and TCP or QUIC underneath. “Layer 7” does not replace the lower layers; it means the proxy understands the application protocol well enough to make an application-aware decision. Conversely, a Layer 4 product can terminate TLS without becoming an HTTP-aware router if it still does not inspect HTTP content.

Strategies of Load Balancing

Static algorithms use configured topology and weights; dynamic algorithms use live connection, latency, queue, or resource signals.

Static Load Balancing Algorithms

Round-Robin

  • Description. Select eligible servers in sequence and wrap to the beginning, producing roughly equal request counts.
  • Use it when. Backends have similar capacity, request cost and duration are reasonably uniform, session affinity is unnecessary, and the simplicity of predictable distribution is valuable.
  • Limitations. It ignores active work, request cost, capacity, and latency, so equal counts can create unequal load.
  • Exact failure case. A receives a 60-second report while B and C receive 20-millisecond lookups. The next long report returns to busy A although B and C are idle.

Weighted Round-Robin

  • Description. Assign fixed capacity weights; weight 4 receives about four times the requests of weight 1.
  • Use it when. Backends have known, stable, unequal capacity—for example, a 16-vCPU server beside a 4-vCPU server—and request costs are broadly similar.
  • Limitations. Static weights cannot reflect noisy servers, slow dependencies, or changing request classes.
  • Exact failure case. CPU-based weights send 80 of 100 database-heavy requests to A, yet A and B each have only 20 database connections. A queues 60 while B has no queue.

IP-Hash

  • Description. Hash a client key, usually its source IP, to select a backend and provide basic affinity. Consistent hashing reduces remapping when membership changes.
  • Use it when. A legacy stateful application needs best-effort affinity and a better application-level affinity key or shared session store is unavailable.
  • Limitations. NAT and proxies make many users share an IP; mobile and IPv6 addresses can change. Pool changes remap clients, and hot clients remain hot.
  • Exact failure case. Ten thousand employees sharing one corporate proxy hash to server B while the other nine servers receive ordinary traffic.

Dynamic Load Balancing Algorithms

Dynamic algorithms depend on delayed, noisy measurements. Production designs also need tie-breaking, slow start, and outlier ejection.

Least-Connections Algorithm

  • Description. Select the eligible server with the fewest active connections, using round-robin to break ties.
  • Use it when. Connections vary substantially in duration, backends have similar capacity, and active connection count correlates reasonably well with work. It is often better than round-robin for long-lived TCP connections.
  • Limitations. Connections differ in cost; HTTP/2 multiplexing and reuse further weaken the connection-to-work relationship.
  • Exact failure case. A has 10 idle WebSockets; B has one CPU-heavy image transform. The next transform goes to saturated B because 1 < 10.

Weighted Least Connections Algorithm

  • Description. Divide active connections by capacity weight and select the lowest score.
  • Use it when. Connection durations vary and backend capacity is predictably unequal.
  • Limitations. It inherits the weaknesses of connection counts and fixed weights; the chosen weight may not represent the current bottleneck.
  • Exact failure case. CPU weights send 64 of 72 uploads to A, but A and B have equal disk IOPS. A's disk queue grows despite a balanced score.

Least Response Time Algorithm

  • Description. Send work to the backend with the lowest moving latency estimate, often combined with active connections.
  • Use it when. Latency differs across otherwise compatible backends and recent latency predicts the cost of upcoming requests—for example, replicas in different locations or with different cache warmth.
  • Limitations. Historical latency may not predict another route; sparse traffic creates stale estimates and hides recovery.
  • Exact failure case. Cached reads give A a 15-ms score, but /annual-report takes five seconds there and 500 ms on B, which owns the shard. The global score chooses A.

Resource-based Algorithm

  • Description. Combine CPU, memory, queue, disk, or worker-slot telemetry into a headroom score.
  • Use it when. Requests consume a known scarce resource, connection count is a poor proxy, and the platform can collect timely, trustworthy metrics. Batch inference, transcoding, and heterogeneous compute pools are common examples.
  • Limitations. Metrics are delayed, require normalization, and can cause herd behavior. Host metrics may also miss the true bottleneck.
  • Exact failure case. A reports 10% CPU, then receives a batch job one second later. Until the next 10-second sample, stale telemetry sends new requests to overloaded A.
No algorithm eliminates the need for admission control, timeouts, backpressure, autoscaling, and capacity testing. A dynamic algorithm can distribute overload more intelligently, but it cannot manufacture capacity.

Load Balancers on AWS

Elastic Load Balancing (ELB) is the AWS service family. It creates managed load-balancer nodes in enabled Availability Zones, monitors registered targets, and routes only to healthy targets. AWS describes the common model in  How Elastic Load Balancing works .
NLB and ALB are regional resources tied to one VPC, with nodes in selected AZ subnets. Use targets in multiple AZs for resilience. ALBs normally require two AZs; NLBs can use one or more. ALB cross-zone balancing is enabled, while NLB cross-zone balancing is disabled by default.
NLB routes connections and flows to instances, IP addresses, or one ALB. ALB interprets HTTP requests and can route to compute, container, Kubernetes, private-IP, and Lambda targets.

NLB: Network Load Balancer

Why NLB is a Layer 4 load balancer

An NLB selects targets using protocols, addresses, ports, and flow state; it cannot route by HTTP URL or header. TLS termination does not make it application-aware. AWS's  NLB target-group documentation  lists TCP, TLS, UDP, TCP_UDP, QUIC, and TCP_QUIC combinations.

Service dependencies

For a functional NLB, provision or select:
  • A VPC and subnet mapping. Select one subnet in every AZ to enable. For an internet-facing NLB, those subnets need a route to an internet gateway. Optional Elastic IP allocations provide a stable public IPv4 address per enabled AZ.
  • Targets. Depending on target type, these can be EC2 instance IDs, private IP addresses, or one Application Load Balancer.
  • A target group. It defines target type, target protocol and port, IP address type, and health-check settings.
  • A listener. It accepts a protocol and port and forwards to the target group.
  • Network controls. Allow data and health checks in NACLs and target security groups. Attach an NLB security group at creation because it cannot be added later. QUIC and TCP_QUIC require an IPv4 NLB without security groups. See  NLB security-group considerations .
  • A certificate for TLS termination. Use an ACM or IAM server certificate when configuring a TLS listener. It is not required for TCP pass-through.
  • IAM permissions. The operator or deployment role needs permission to create and configure ELBv2 resources and related service-linked roles.
  • Route 53, Global Accelerator, Auto Scaling, ECS, or EKS integrations are optional—not prerequisites for the NLB resource itself.

Service scope

An NLB is regional and tied to one VPC. Its nodes and IP addresses are zonal: each selected subnet enables one AZ and receives an NLB network interface/address. That combination explains why it is inaccurate to call the NLB itself AZ-scoped. Multi-AZ is a configuration of one regional load balancer.

Traffic type routed

  • TCP: pass a TCP byte stream to a TCP-compatible target group.
  • TLS: terminate TLS at the NLB and open TCP or TLS to targets, depending on target-group configuration.
  • UDP and TCP_UDP: balance datagram flows or both protocol types on the same port.
  • QUIC and TCP_QUIC: support QUIC flows or combined TCP and QUIC configurations on an IPv4 NLB without associated security groups.
An NLB is suitable for HTTP or HTTPS only when treating it as TCP/TLS traffic is sufficient. It cannot send /checkout and /catalog to different target groups by inspecting the path.

Compatible services and targets

  • Amazon EC2 and Auto Scaling: register instances or their private IP addresses; Auto Scaling can maintain target registration.
  • Amazon ECS: route to container instances or directly to task IPs in awsvpc mode.
  • Amazon EKS: route to nodes or Pod IPs using the AWS Load Balancer Controller. AWS documents  NLB routing for EKS .
  • Private IP endpoints: register reachable private IPs, including supported addresses reached through VPC peering, Transit Gateway, Direct Connect, or Site-to-Site VPN. Publicly routable IPs cannot be NLB IP targets.
  • Application Load Balancer: an NLB target group can contain one ALB, useful when static IP/PrivateLink-style entry and Layer 7 routing are both needed.
  • AWS PrivateLink: an NLB can front an endpoint service consumed through interface VPC endpoints.
  • NLB does not support Lambda as a target type.

Steps to provision and configure an NLB

  • Design the path. Choose the Region, VPC, internet-facing or internal scheme, IP address type, enabled AZs, and whether cross-zone balancing is required.
  • Prepare networking. Select one subnet per enabled AZ. Configure route tables, NACLs, and target security groups. For TCP, TLS, and UDP families, preferably attach an NLB security group at creation; for QUIC or TCP_QUIC, use IPv4 and do not associate one. Reserve EIPs if fixed public IPv4 addresses are required.
  • Create a target group. Select instance, ip, or alb; choose the target protocol and port; configure TCP, HTTP, or HTTPS health checks as supported.
  • Register targets. Put at least one healthy target in each enabled AZ. Ensure the target listens on the target and health-check ports.
  • Create the NLB. Choose Network Load Balancer, scheme, VPC, subnet mappings, addresses, and security groups.
  • Create a listener. Select TCP, TLS, UDP, TCP_UDP, QUIC, or TCP_QUIC and forward to the target group. For TLS, attach a certificate and security policy.
  • Tune behavior. Decide on cross-zone load balancing, client IP preservation, deregistration delay, stickiness where supported, and zonal health behavior.
  • Validate. Wait for healthy targets, test the NLB DNS name from an allowed client, confirm source-IP expectations, and inspect CloudWatch metrics and access/flow logs as appropriate.
  • Publish DNS. Optionally create a Route 53 alias record or place Global Accelerator in front.
AWS provides console, CLI, and infrastructure-as-code details in  Create a Network Load Balancer  and  Create an NLB target group .

Costing

NLB cost has two primary ELB components:
AWS bills each hour or partial hour the NLB runs, plus Network Load Balancer Capacity Units (NLCUs) used per minute. An NLCU measures three dimensions—new connections or flows, active connections or flows, and processed bytes—and the highest normalized dimension determines usage for the period. The allowance differs for TCP, TLS, and UDP. Capacity reservations, standard data transfer, public IPv4 addresses, and other connected services can add charges.
Rates vary by Region and can change, so use the official  Elastic Load Balancing pricing page  and  AWS Pricing Calculator  instead of copying a rate into long-lived architecture documentation.

ALB: Application Load Balancer

Why ALB is a Layer 7 load balancer

An ALB terminates the client-side HTTP conversation and can choose an action using application content. Listener rules can match host name, path, HTTP header, method, query string, and source IP. Actions can forward to one or more target groups, redirect, return a fixed response, or perform supported authentication. That requires understanding HTTP semantics, which makes ALB a Layer 7 proxy.

Service dependencies

For a functional regional ALB, provision or select:
  • A VPC and subnets. A standard regional ALB requires subnets in at least two Availability Zones. Internet-facing ALBs use public subnets with an internet-gateway path; targets can remain in private subnets.
  • An ALB security group. Allow client traffic to listener ports and outbound traffic to target and health-check ports. Target security groups should allow inbound traffic from the ALB security group rather than the whole internet.
  • Targets and target groups. Target type can be instance, private IP, or Lambda. Instance/IP target groups define HTTP or HTTPS ports and health checks. A Lambda target group contains one function.
  • A listener and rules. HTTP or HTTPS listeners need a default action; additional ordered rules are optional.
  • IAM permissions. Deployment identities need ELBv2 and related permissions. Lambda targets also need resource-based permission allowing the load balancer to invoke the function.
  • AWS WAF, Amazon Cognito or OIDC authentication, Route 53, Global Accelerator, Auto Scaling, ECS, and EKS integrations are optional.

Service scope

Like NLB, ALB is a regional resource associated with one VPC and implemented by nodes in enabled AZs. It is neither VPC-global nor tied to one AZ. A standard ALB enables at least two AZs. Cross-zone distribution is enabled at the load-balancer level; target-group settings can alter the scope used by that group.

Traffic type routed

  • ALB listeners accept HTTP and HTTPS. Depending on listener and target-group configuration, ALB supports HTTP/1.1, HTTP/2, WebSocket upgrades, and gRPC. AWS documents WebSocket and HTTP/2 behavior in  ALB listeners , and documents backend HTTP/2/gRPC combinations in  ALB target groups .
ALB does not route arbitrary TCP or UDP protocols. An HTTPS listener terminates TLS at the ALB; it is not opaque TLS pass-through.

Compatible services and targets

  • Amazon EC2 and Auto Scaling: register instance IDs or private IPs; Auto Scaling manages registration during scale events.
  • Amazon ECS: route to services on EC2 or Fargate. Tasks using awsvpc network mode use the ip target type. See  Using an ALB with ECS .
  • Amazon EKS: the AWS Load Balancer Controller can create ALBs for Kubernetes Ingress or Gateway resources and target nodes or Pods.
  • AWS Lambda: an ALB can invoke one Lambda function registered in a Lambda target group. The request is converted to an event and the function returns an ALB-shaped response. See  Lambda functions as ALB targets .
  • Private IP endpoints and on-premises services: use reachable private IP targets within the documented private ranges, including connections through peering, Transit Gateway, Direct Connect, or VPN.

Types of traffic routable

Listener rules can route using:
  • host, such as admin.example.com versus api.example.com;
  • path, such as /orders/* versus /catalog/*;
  • HTTP method, header, or query string;
  • source IP;
  • weighted forwarding to several target groups for gradual releases; and
  • a default action when no higher-priority rule matches.
ALB can also redirect HTTP to HTTPS, return a fixed response without contacting a target, authenticate through OIDC or Amazon Cognito, and integrate with AWS WAF. WebSocket routing becomes persistent after the HTTP upgrade, so later messages on that connection are not independently rebalanced.

Steps to provision and configure an ALB

  • Design the HTTP boundary. Choose the Region, VPC, internal or internet-facing scheme, IP address type, host names, routes, TLS policy, and target types.
  • Prepare networking. Select subnets in at least two AZs, configure routes and NACLs, create an ALB security group, and restrict each target security group to the ALB security group.
  • Create target groups. Choose instance, ip, or lambda; for network targets select HTTP/HTTPS, port, protocol version, and health-check path and matcher.
  • Register targets. Register instances/IPs or one Lambda function and wait for health checks. Deploy capacity in every enabled AZ where practical.
  • Create the ALB. Select Application Load Balancer, scheme, IP address type, VPC, at least two subnets, and security groups.
  • Create listeners. Commonly, add HTTP port 80 with a redirect to HTTPS and HTTPS port 443 with an ACM certificate and current security policy.
  • Add ordered rules. Define specific host/path/header matches before the default rule. Forward, redirect, authenticate, or return fixed responses as required.
  • Integrate safeguards. Optionally attach AWS WAF, enable access logs to S3, set deletion protection, and review idle timeout, desync mitigation, and header settings.
  • Validate. Confirm target health, test every listener rule and redirect, verify X-Forwarded-* handling, and inspect CloudWatch request, target, error, and latency metrics.
  • Publish DNS. Create a Route 53 alias for the ALB DNS name and verify certificate host-name coverage.
AWS provides a detailed console and CLI walkthrough in  Create an Application Load Balancer .

Costing

ALB cost has two primary ELB components:
AWS bills each hour or partial hour the ALB runs, plus Load Balancer Capacity Units (LCUs) used per minute. An ALB LCU measures four dimensions: new connections, active connections, processed bytes, and rule evaluations. The highest normalized dimension determines LCU usage. Mutual TLS, LCU reservation, standard data transfer, public IPv4 addresses, AWS WAF, authentication-related services, and other integrations may add charges.
Use the current  official ELB pricing page  and  AWS Pricing Calculator . For cost estimation, model a realistic peak hour: connection reuse affects new-connection usage, large responses affect processed bytes, and large rule sets affect rule evaluations.

Build Your Own Load Balancer

This lab runs three private FastAPI replicas behind an Nginx reverse proxy. Docker Swarm supplies replication, service discovery, and an internal overlay network; Nginx applies  least_conn ; k6 generates requests with different durations.
 Download the complete runnable project . It contains every source file, the full Nginx configuration, the Swarm stack, the k6 test, and operating instructions.
This is a learning system, not a production edge. Production deployments need multiple proxy replicas, TLS, monitoring, image registries, and an external mechanism for reaching healthy proxy nodes.

Architecture


Only Nginx publishes a port. FastAPI tasks remain on the private  backend  network and are reachable through Swarm DNS as  tasks.api .

1. Simulate work with FastAPI

The  /work  endpoint waits asynchronously, then returns its container hostname.  asyncio.sleep()  simulates I/O wait without blocking the event loop; it does not simulate CPU saturation.
INSTANCE_ID = socket.gethostname()
@app.get("/work")
async def work(response: Response,
delay_ms: int = Query(200, ge=0, le=10_000)):
started = time.perf_counter()
await asyncio.sleep(delay_ms / 1_000)
response.headers["X-Backend-Instance"] = INSTANCE_ID
return {
"instance": INSTANCE_ID,
"requested_delay_ms": delay_ms,
"elapsed_ms": round((time.perf_counter() - started) * 1_000),
}
The repository includes  /health , pinned dependencies, and a Dockerfile that launches Uvicorn on port  8000 .

2. Configure Nginx

Nginx resolves every FastAPI task through Docker's embedded DNS and selects the upstream with the fewest active connections:
resolver 127.0.0.11 valid=5s ipv6=off;
upstream api_pool {
zone api_pool 64k;
least_conn;
server tasks.api:8000 resolve max_fails=2 fail_timeout=5s;
}
location / {
proxy_pass http://api_pool;
proxy_connect_timeout 1s;
proxy_read_timeout 3s;
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
}
 resolve  refreshes task addresses as replicas change.  max_fails  and  fail_timeout  provide passive failure handling. A four-second  /work  request exceeds the three-second read timeout and may be retried once, illustrating why attempt timeouts differ from end-to-end deadlines.

3. Isolate the API with Docker Swarm

The stack creates three DNS-round-robin API replicas. Only Nginx publishes a host port:
services:
api:
image: lb-lab-api:1.0
networks: [backend]
deploy:
replicas: 3
endpoint_mode: dnsrr
nginx:
image: nginx:1.30.4-alpine
ports:
- target: 80
published: 8080
mode: ingress
networks: [backend]
networks:
backend:
driver: overlay
internal: true
The full stack also defines health checks, restart policies, resource limits, and a read-only Swarm config for  nginx.conf .

4. Build and deploy

From the repository root:
docker swarm init
docker build -t lb-lab-api:1.0 ./app
docker stack config -c stack.yml
docker stack deploy --resolve-image never -c stack.yml lb-lab
docker stack services lb-lab
The local image is sufficient for a one-node Swarm. For multiple nodes, push it to a registry reachable by every node.

5. Test with k6

 Grafana k6  runs JavaScript-defined virtual users and aggregates throughput, failures, and latency. Scenarios control load; checks record correctness; thresholds determine the test's exit status.
export const options = {
scenarios: {
variable_work: {
executor: 'ramping-vus',
stages: [
{ duration: '15s', target: 10 },
{ duration: '30s', target: 20 },
{ duration: '15s', target: 0 },
],
},
},
thresholds: {
checks: ['rate>0.99'],
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<3000'],
},
};
export default function () {
const delay = [50, 100, 250, 500, 1000]
[Math.floor(Math.random() * 5)];
const response = http.get(`${BASE_URL}/work?delay_ms=${delay}`);
check(response, { 'status is 200': (r) => r.status === 200 });
sleep(0.2);
}
Run it locally with  k6 run ./k6/load-test.js , or use the container command documented in the repository.

6. Monitor the result

Follow proxy logs with  docker service logs -f lb-lab_nginx . Each line reports the upstream address and timing, proving that multiple tasks receive traffic. During a test, run  docker service scale lb-lab_api=5 ; Nginx discovers the new task addresses within a few DNS refresh intervals.
Evaluate the result by checking correctness,  http_req_failed , latency percentiles, and backend distribution. Least-connections need not produce identical request counts when request durations differ.
Clean up with  docker stack rm lb-lab . If Swarm was enabled only for this lab, run  docker swarm leave --force  afterward.

Closing Remarks

AWS NLB and ALB provide Layer 4 and Layer 7 load balancing. The Nginx/Swarm lab reveals major choices: public/private boundaries, backend discovery, load measurements, retries, and success criteria. These principles scale from a laptop to a regional production system.

Series: Distributed Systems and Architecture (1 article)

  1. 1
    Load Balancing in Distributed Systems This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article