AWS Transform MGN stuck at Initiating: private endpoints behind a corporate proxy
An AWS Transform MGN agent registers, reports Healthy, and never starts replicating. The Python installer honours no_proxy, but the JVM-based replication component only reads the http.nonProxyHosts system property, so it proxies traffic bound for a private interface endpoint and the TLS handshake hangs. How to narrow it to the transport layer from the agent log, and how to inject the property via JAVA_TOOL_OPTIONS.
Intro
Rehosting a Windows server with AWS Transform MGN is about as well-trodden as migration work gets. Install the agent, wait for the initial sync, launch a test instance. The documentation is good, the service has been around for years, and on a normal day the hardest decision is which instance type to pick.
Our case was more complex...
Together with my AWS colleagues Marc Schaepers (Sr Delivery Consultant, DevOps) and Michael Daehnert (Sr Delivery Consultant), we needed to migrate a handful of Windows machines for an enterprise customer. Nothing exotic, nothing that should have taken more than an afternoon.
It was. First machine. The agent installed cleanly, the source server reported
Healthy in the console, and then nothing happened... for 19 hours... Replication progress: 0%. Replication initiation steps: not one of them completed. CloudTrail: silent.The Problem
So what do you do when a service that works everywhere refuses to work here?
We spent some time looking in the wrong places, and the reason for that is worth more than the fix itself. The console said the server was healthy. The lifecycle said it was initiating. There was no error anywhere in the console, and no failed API call in CloudTrail — because the service had never been asked to do anything at all.
What we were actually looking at was not a migration problem. It was two perfectly reasonable network controls, owned by two different teams, quietly cancelling each other out, with an agent caught in between and no way to tell us so.
The Problem
There are two established ways for a server in a corporate data centre to reach an AWS service.
The first is through the corporate proxy to the public service endpoint. This is what most proxied enterprises do, and it is what the agent FAQ describes. The second is over Direct Connect or a VPN to an interface VPC endpoint, so the traffic never touches the internet at all. That is what Installing the Agent on a secured network and the matching prescriptive guidance describe.
Both are solid. Both are documented. Both work.
What we have not found described is having both in the path for the same traffic — and that turns out to be surprisingly easy to arrive at. The proxy policy is usually blanket: applied centrally to every server, no exception for internal destinations, owned by the network team. The private endpoint is built separately by the cloud team, for perfectly sound security reasons. Neither team did anything wrong. Neither can easily unwind their decision. And the collision stays invisible until an agent stops making progress.
The mechanics, once you see them, are almost disappointingly simple. The agent is pointed at a private RFC 1918 address inside the customer's VPC, and simultaneously told to send everything through a proxy that sits on the internet side and has no route there. It uses the proxy. The connection waits. Forever.
What the Log Actually Said
The agent log on the source server is the only place the real error appears, and it had been there for nineteen hours: 204 identical failures, every 90 seconds.
1
2
3
ERROR agentHandshake failed: unable to attain a replicator
Caused by: javax.net.ssl.SSLException: Read timed out
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)It dies in
startHandshake, meaning the TLS handshake never completed and no HTTP request was ever sent — which rules out credentials, IAM, endpoint policies and certificate interception in one go, since all of those are evaluated after TLS.Why the Two Halves Behaved Differently
Something in the picture still didn't add up. Registration had worked. The server was in the console. If the proxy was eating everything, how did the agent ever get registered in the first place?
Because the agent is not one program. It is two, and they handle proxies differently.
The installer is Python, and it reads
http_proxy, https_proxy and no_proxy exactly as you would expect. That is why registration completed on the first attempt, and why the server showed up Healthy.The replication component runs on the JVM. It picks the proxy up from the environment and logs it at service start:
1
Proxy info is: ProxyParams{host='proxy.internal.example', port='80'}Neither
no_proxy nor nonProxyHosts appears anywhere in the log. And that is the whole story: Java does not read the no_proxy environment variable natively. It reads the http.nonProxyHosts system property. So the replication component dutifully handed requests for a private 10.x address to an internet proxy, and the customer's carefully configured no_proxy entry was never consulted.Which also explains why setting
no_proxy machine-wide and reinstalling the agent changed nothing. We were configuring the half that already worked.The Fix
1
2
3
4
5
[System.Environment]::SetEnvironmentVariable("JAVA_TOOL_OPTIONS",
"-Dhttp.nonProxyHosts=vpce-xxxxx.mgn.eu-central-1.vpce.amazonaws.com|10.x.y.*|169.254.169.254|localhost|127.*|[::1]",
"Machine")
Restart-Service <agent service name> -ForceJAVA_TOOL_OPTIONS is prepended to the JVM's arguments at process start, so the replication component picks up http.nonProxyHosts without any other change to the agent, the installer, or the service.All eight replication initiation steps went green within minutes.
Three things worth knowing if you use this:
- No reinstall needed. The proxy is read at service start, not baked in at install time, so restarting the service is enough.
http.nonProxyHostsgoverns HTTPS too. There is no separatehttps.nonProxyHostsproperty.- Setting it replaces the JVM default of
localhost|127.*|[::1], so keep those in your list and add the instance metadata address.
Also relevant on Windows:
--proxy-address is a Linux installer parameter only, so environment variables are the mechanism available to you.The Next Wall: Port 1500
Fixing the control plane does not finish the job, it relocates the failure. Our next error was
Failed to connect AWS Replication Agent to Replication Server, with every preceding initiation step green and a replication server happily running in the staging subnet.This is the data path, and the default setting assumes internet connectivity:
By default, data is sent from the source servers to the replication servers over the public internet, using the public IP that was automatically assigned to the replication servers.
On a server whose only outbound route is a proxy, that cannot work — and the documentation is explicit that a web proxy cannot carry port 1500 under any circumstances. Use private IP for data replication has to be selected in the replication template, and it applies only to newly added source servers, so set it before you add anything.
What To Set Up Front
If you are deploying MGN into a proxied enterprise with private endpoints, treat all of this as part of the standard build rather than as troubleshooting:
JAVA_TOOL_OPTIONSwithnonProxyHosts, set before installing, alongside the usual proxy variables. This is the one nobody expects.- Use private IP for data replication in the replication template, selected before any source servers are added.
- An S3 interface endpoint and
--s3-endpointat install. It is part of the documented secured-network pattern, and both the agent and the replication servers pull their binaries from S3. - The
CreateSessionForMgnstatement in the VPC endpoint policy, if that policy is scoped down at all:
1
2
3
4
5
6
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:<region>:*:*/POST/CreateSessionForMgn"
}That is the API behind the credential exchange. A missing statement gives you a 403 rather than a hang, but it stops you at exactly the same step.
- Bring the proxy team in early and point them at the connectivity requirements, which include an explicit note that SSL interception should not be applied between source servers, or replication servers, and the MGN API endpoint. That holds whether you end up on the private endpoint or the public one.
Closing Thought
The fix was one environment variable. Finding it took most of two days, and nearly all of that time went into AWS-side resources that had never been created, chasing a failure that was fully described in the first few lines of an exception in a log file on the source server.
So the real lesson is not about proxies. It is about where the truth lives. When the MGN console reports Healthy and the lifecycle shows Initiating with no error surfaced, the console is telling you what it knows, which in this case was almost nothing. The agent knew exactly what was wrong, had known for nineteen hours, and was writing it down every ninety seconds.
Read the agent log first.
It took a mix of people from the AWS team to get here: Marc Schaepers (Sr Delivery Consultant, DevOps), Michael Daehnert (Sr Delivery Consultant) and Sergii Shchaslyvyi (author) to test the solution.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article