AWS Network: Where having two VPNs, two DX, or hybrid connections can fail.
Two connections out of AWS, with two Customer Gateways, two different routers, two exit points pointing to the same VGW. On paper, that's redundancy, since you've got a pair of everything, and it's a design any infra team sets up to sleep easy at night. Except there's one variable that actually decides whether that redundancy holds up, and it's not the number of connections, it's the routing intelligence behind them. In this article, we'll dig into how BGP works and how it can save your weekend.
Series: AWS Network (3 articles)
- 3AWS Network: Where having two VPNs, two DX, or hybrid connections can fail. This article
BGP: The Protocol That Refused to Die
If you followed the previous article in this networking journey, about how Cisco intuition can trip you up on AWS, you already know the thread we planted there: on AWS, routing isn't discovered, it's declared. You write the route, it exists, with no protocol deciding anything behind the scenes.
BGP is the exception to that rule, which is exactly why it confuses so many people. It's the only place in AWS networking where the path goes back to being genuinely discovered, by protocol, negotiated between endpoints, instead of typed in by you. After getting used to the idea that every route on AWS is a deliberate act, watching BGP show up and decide the path on its own breaks the exact intuition you just rebuilt.
But let's start from the beginning: what BGP is and how it works.
(translate by IA)
Every network has an owner, and that owner has a number
Think about international mail (yes, I'm reusing the same example from the SWIFT article). Every country has its own postal service, handles its own internal distribution however it wants, and decides which countries it strikes deals with to exchange mail with the rest of the world. On the internet, that "country with its own postal service" is called an Autonomous System, or AS. It's any network with its own administration and a unique identifying number, the ASN (Autonomous System Number).
Your company can have an AS, just like an internet provider can, and AWS also shows up on the other side as an AS whenever you open a BGP session with it, with its own ASN (64512 by default, unless you pick a different one when creating the Virtual Private Gateway). In our lab, two more ASes enter the picture: each FRR instance representing a different on-premises point, each with its own ASN.
How a route "knows" where it's already been
Now, the problem BGP solves: how does an AS know a path exists to a network on the other side of the world, if it's never talked to it directly?
Picture a letter leaving Brazil for Japan, passing through Panama, the US, and Dublin before reaching its destination. Every country that receives that letter stamps its own mark on it before passing it along, so by the time it reaches Dublin, you can see exactly where it's already been: Brazil, Panama, the US. That's enough for any country along the way to know the origin and the route, without needing to know the entire map all the way to the final destination.
Now imagine if the postal system worked with the same autonomous decision logic BGP has: every country, on receiving a letter, decided on its own whether to pass it along or not. In this hypothetical scenario, if Ireland saw its own stamp already on a letter it's receiving again, it would know right away that this letter had already passed through there before, in some earlier cycle, and would refuse it, because passing it along again would just make that letter loop forever. That instinct of "I've already seen my own stamp here, something is going in circles, and I'm stopping it now" is exactly what BGP has built in for real, just automatic and in milliseconds.
The technical name for this mechanism is AS-path loop prevention: every AS, when announcing a route, includes the list of all the ASNs it has already passed through (the AS-path), and if a BGP router receives an announcement that already contains its own ASN on that list, it discards the announcement right away. That's how BGP avoids routing loops without needing any central coordination.
Hold onto that AS-path detail, because it comes back twice: later in this same article, when the lab's two FRR instances announce the same network and AWS has to decide which path to prefer, and again in Article 2, when I deliberately manipulate that attribute with AS-path prepending.
Which metrics does AWS actually use to decide which route to prefer?
BGP doesn't pick a route based on "which one's faster" or "which one has lower latency." It chooses by following a priority order between attributes, and three of them are the ones you'll actually touch when working with AWS:
- AS-Path length: the shorter the path (fewer ASes in between), the more preferred. This is the criterion that decides the tie in our lab: both FRR instances are a single hop away from AWS, so both have the same AS-Path length.
- Local Preference: sets the outbound preference inside your own AS, for traffic heading toward AWS. It's not something AWS applies, it's something you configure on your side.
- MED (Multi-Exit Discriminator): used when you have more than one path announcing the same prefix to AWS, signaling which connection is preferred for traffic coming in from AWS. Worth an important heads-up here, because it connects directly with the limitation this article's simulation carries: in a real-world scenario where Direct Connect and VPN coexist, AWS prefers Direct Connect automatically, because Direct Connect routes arrive with an implicitly lower MED than VPN routes. That behavior doesn't show up for free in our lab, because the two connections we're setting up are technically both VPN, so there's no automatic preference between them. Deliberately manipulating MED to reproduce that kind of preference is a topic for Article 3.
In the lab, both FRR instances announce exactly the same network, with the same AS-Path length. That means that when both paths are healthy at the same time, which one AWS actually uses isn't obvious ahead of time, and documenting what really happens in the console is part of this article's value.
All the theory so far is universal, it works for any network that speaks BGP, inside or outside the cloud. But on AWS specifically, this protocol shows up in three places: VPN Site-to-Site, Direct Connect, and Transit Gateway. In all three, the principle is the same (AS, ASN, path vector, route propagation), only the underlying transport changes. This article goes deep on the VPN scenario because it's the most accessible way to set up a lab with minimal infrastructure and watch the concept run for real, but keep this in mind: the vocabulary you learn here comes back exactly the same way in Article 2 (Direct Connect) and Article 5 (Transit Gateway).
Whenever you speak BGP with AWS, AWS is always one of the two endpoints of the conversation, never a neutral middleman just carrying your route without getting involved. Going back to the postal analogy, it's not like AWS is a delivery truck that just carries the letter without opening it, it's a real country in this analogy, with its own AS, deciding what it accepts and what it decides to announce back.
The AS on AWS's side: the Virtual Private Gateway
In the VPN scenario, the one representing AWS in the BGP conversation is the VGW (Virtual Private Gateway), which has its own ASN and gets the value 64512 by default unless you pick a different one when creating the VGW. On the other side, each on-premises router is represented by a Customer Gateway, with an ASN you choose.
Nerd-mode detour: inside AWS, the Customer Gateway is just a configuration object, a record saying "this is the public IP and the ASN of this on-premises router." It doesn't route anything by itself. The actual routing happens on your physical or virtual device (each FRR, in the lab). People mix the two up all the time, treating the Customer Gateway as if it were the router itself, when really it's just its registration card inside AWS.
And here's a rule that sounds like bureaucratic detail but in practice avoids the first mistake everyone makes: the VGW's ASN and each Customer Gateway's ASN need to be different from each other. If you use the same number, BGP identifies that as the same route coming back to itself, the loop prevention mechanism we saw earlier kicks in, and the route gets discarded.
Nothing stops you from creating two, three, four different Customer Gateways, all attached to the same VGW, each representing a distinct on-premises router, or even a combination of VPN with Direct Connect, and that's exactly the design infra teams use to build redundancy: if one path goes down, the other takes over.
What AWS's documentation makes clear is that this "the other one takes over" depends on BGP to actually work. In AWS's own words, describing this redundancy scenario: the Customer Gateways should announce the same network to the VGW, and it's BGP that decides which path to use and automatically redirects traffic if one of them fails. Static routing doesn't enter that conversation the same way, because there's no live signaling saying "this path is dead, use the other one."
From learned route to usable route, or propagation
Knowing that AWS learned a route via BGP isn't the same thing as that route already being available for your VPC to use. There's an intermediate step, and it's exactly where most beginners get stuck without understanding why: route propagation.
When the VGW learns a route via BGP, it becomes available to be propagated, but it only actually shows up in your VPC's route table if you've enabled propagation on that specific route table for that specific VGW. It's a separate switch, it doesn't happen automatically just because the BGP session came up.
Two connections, one fake redundancy: the problem static routing doesn't solve
Every time you spin up a VPN Site-to-Site connection on AWS, the console offers two options without much explanation: static routing or dynamic BGP. On its own, each option looks like just a configuration choice. The problem shows up when you have two redundant connections (VPN with VPN, or VPN as a Direct Connect backup) and assume that whichever one you pick, they'll cover for each other. They won't, unless it's BGP.
The easy path is static routing
You manually type the destination CIDR straight into the route table, pointing to a specific connection, and that comes up fast, without requiring any protocol knowledge, working fine for the case of a single connection.
The real problem shows up in the scenario of two redundant connections: static routing has no way of knowing, in real time, that one connection died and that traffic should migrate to the other. There's no live signal exchange between the two sides saying "this path no longer exists, prefer the other one." And if the on-premises topology changes too, someone has to remember to update the route by hand, no exceptions.
The right path (and a much cooler one) is dynamic BGP
Each end announces the same network via BGP, with its own ASN, AWS learns both paths and picks between them using the attributes we saw before (AS-Path, and the others when applicable). If a BGP session drops because the router on that side went unavailable, the corresponding route is withdrawn automatically, and AWS already has the other path learned and ready to keep using. There's no manual intervention in this process.
The decision yardstick
Static routing is acceptable when you have a single connection, a fixed scenario, low criticality, or when the on-premises device simply doesn't support BGP. The moment you decide to have two connections for redundancy, static routing stops making sense as a high-availability strategy, because the redundancy it delivers is just having two paths exist, not having a system that knows how to switch between them on its own.
Everything said so far is just talk until you see, with your own eyes, the difference between a static connection that doesn't migrate on its own and a BGP connection that does, and that's what the lab is here to show.
Target infrastructure: 1 VPC, 2 EC2 instances running FRR (FRR-A and FRR-B, each representing a different on-premises point), 1 real VGW attached to that same VPC, 2 Customer Gateways, 2 VPN Connections. All the networking and compute infrastructure goes up via CloudFormation, reusing the same two modules that will show up again in the series' next articles, with the second module instantiated twice.
An honest note on the simulation: ideally, this lab would represent Direct Connect and VPN coexisting, which is the most common redundancy scenario in a real environment. The problem is that Direct Connect requires a physical circuit and provisioning through a partner, and isn't something you can spin up in a personal lab. That's why both connections in this lab are technically VPN Site-to-Site. FRR-A plays the narrative role of "what would be Direct Connect" and FRR-B plays the actual VPN, just to set the scene the series is building toward, but the technical behavior of both is identical: same connection type, same AS-Path length, with none of the automatic preference real Direct Connect would have over VPN. What this lab proves is the mechanics of BGP convergence when a path goes down, not Direct Connect's specific preference, which will only show up for real when we deliberately manipulate MED in Article 3.
The lab, step by step
Module 1: modules/vpc-base.yaml
The network base that every upcoming article will import (via
Fn::ImportValue or nested stack), containing a VPC, public subnet, Internet Gateway, and route table, unchanged from the original module.Technical checkpoint 1: confirm the VPC and subnet were created, route table associated, no VPN route yet. This is the baseline.
Module 2: modules/frr-router.yaml (nested stack, instantiated twice)
The same module from the original design, unchanged in the template, except now it goes up twice, as two separate stacks (
frr-a and frr-b), each with a different value for FrrAsn, keeping the access guardrail via SSM Session Manager, with no SSH exposed.Technical checkpoint 2: via Session Manager,
systemctl status frr confirming the daemon is running on FRR-A and FRR-B separately, before moving on to the next step.Lab sequence
- Deploy
vpc-base.yaml. (Checkpoint 1, already described above.) - Deploy
frr-router.yamlas stack FRR-A (narrative role of Direct Connect),FrrAsn = 65001. (Checkpoint 2.) - Deploy
frr-router.yamlagain, as stack FRR-B (the actual VPN),FrrAsn = 65002, independent of the first one. (Checkpoint 2, repeated for FRR-B.) - Create the VGW with ASN 64512 (Amazon's default) and attach it to the VPC.
Technical checkpoint 3: VGW in "attached" state. - Create Customer Gateway A (FRR-A's public IP, ASN 65001) and Customer Gateway B (FRR-B's public IP, ASN 65002).
Technical checkpoint 4: both Customer Gateways in "available" state.
Phase 1: static routing
- Create Connection A (static) between Customer Gateway A and the VGW, and Connection B (static) between Customer Gateway B and the same VGW, both with the same static destination route:
172.16.0.0/24.
Technical checkpoint 5, document it live: record exactly what AWS allows here. Whether both static routes to the same CIDR coexist in the route table, whether one overwrites the other, or whether the console blocks the second one. This behavior needs to be captured with an actual screenshot during execution, there's no point presuming here what's going to happen without having run it. - Bring up the IPsec tunnels on FRR-A and FRR-B (no BGP session in this phase).
Technical checkpoint 6: confirm which connection the route table is actually using in this initial "healthy" state. - Bring down whichever connection is active (stop the IPsec service or the corresponding FRR instance) and watch the actual behavior for a few minutes.
Technical checkpoint 7, the most important one in the static phase: screenshot the route table before and after the failure, showing whether traffic migrated or stayed pointed at the dead path. This is the record that proves (or rigorously disproves) the limitation described in the theory.
Phase 2: dynamic BGP
- Reconfigure Connection A and Connection B for dynamic BGP. Configure FRR-A and FRR-B to announce the same network
172.16.0.0/24, each with its own ASN.```
! FRR-A
router bgp 65001
neighbor 169.254.X.X remote-as 64512
address-family ipv4 unicast
network 172.16.0.0/24
neighbor 169.254.X.X activate! FRR-B
router bgp 65002
neighbor 169.254.Y.Y remote-as 64512
address-family ipv4 unicast
network 172.16.0.0/24
neighbor 169.254.Y.Y activate - Bring down the same side that was brought down in Phase 1 (same test, now with BGP).
Technical checkpoint 9, the final contrast: screenshot the route table before and after, showing the automatic convergence to the remaining path, with no manual intervention at all. This pair of screenshots (Checkpoint 7 vs. Checkpoint 9) is the central visual argument of the entire article.
Things that can go wrong, and how to fix them
- BGP session stuck in "Active" and never reaching "Established": most of the time this is because the Security Group isn't allowing UDP 500 and 4500 to AWS's public IP, and the IPsec tunnel never actually comes up, even as the BGP session keeps trying to connect on top of a dead tunnel.
- AWS doesn't accept more than 100 routes announced on a single BGP session (it now supports 1,000): if FRR announces more than that, the session drops to "Idle." Summarize your networks before announcing them.
- eBGP multihop disabled by default on AWS: BGP peers need to be directly connected, a single hop. This applies equally to both FRR instances.
- Propagation isn't automatic: the BGP session can be healthy, the route being announced and received, and it still won't show up in the route table, because the propagation toggle for that specific route table was never enabled.
- Two static routes to the same CIDR: this is the central gotcha in Phase 1, and it needs to be documented with the actual behavior observed in the console, not assumed. Whether AWS accepts both and always uses whichever was registered first, or rejects the second one, or does something else entirely, that's what turns into the most valuable paragraph in this section.
- Don't confuse the simulation with real Direct Connect: since both connections in the lab are technically VPN, don't expect to see AWS automatically preferring one over the other. That preference is real in the Direct Connect vs. VPN world, but it requires a physical circuit to exist, and deliberate MED manipulation to be reproduced in a lab, which is saved for Article 3.
When do I NOT use dynamic BGP?
Worth documenting honestly, because not every piece of infrastructure needs this:
- Temporary or PoC VPNs that will be torn down in a matter of days.
- Very low-criticality environments, with a single connection, where nobody's going to touch the topology once it's set up.
- When the on-premises device simply doesn't speak BGP (some legacy firewalls and cheaper appliances only do static routing, no option).
In that last case, it's worth being honest with the reader: if your device doesn't support BGP and you've got two redundant connections anyway, accept that real redundancy is going to depend on manual intervention when something fails, and document that as a known risk, instead of selling leadership on a resilience the architecture doesn't actually deliver (or write some gnarly script to automate it) (or, even better, go yell at your manager that this isn't how it's done).
Conclusion
BGP isn't a VPN feature, it's the layer that decides whether the redundancy you designed on paper actually exists the moment it needs to. Two connections on static routing are two paths that exist, not a system that knows how to switch between them.
This article proved that with the simplest infrastructure possible, two VPNs playing the role of a bigger scenario, and a test anyone can reproduce. In the next article, the same protocol steps into a real Direct Connect, and we start deliberately manipulating which path AWS prefers, with AS-path prepending, instead of just letting BGP converge on its own.
Series: AWS Network (3 articles)
- 3AWS Network: Where having two VPNs, two DX, or hybrid connections can fail. This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article