Using AssumeRoleWithWebIdentity when your provider is on a private network
This article will explain to you how you can use your non-internet accessible OpenID Compliant JWT issuer with AssumeRoleWithWebIdentity from AWS STS without having to consider exposing it to the internet.
Use split-horizon DNS to federate a private OIDC identity provider with AWS STS
In this post, I'll show you how to use AWS Security Token Service (AWS STS)
AssumeRoleWithWebIdentity with an OpenID Connect (OIDC) identity provider (IdP) that is hosted on your private network that is not internet accessible. At the time of writing, AWS STS needs to be able to connect to the issuer and JWKS endpoint of an OpenID provider to get the keys to verify the JWT tokens your provider has issued.
AWS Customers have told us they have providers hosted in private networks, often associated with deployment platforms like Terraform, Jenkins, self-hosted Gitlab and self-hosted Github, or PingFederate/Activate Directory Federation Services (ADFS) that they want to start using with AssumeRoleWithWebIdentity, but do not want to expose those platforms to traffic from the internet.
In this post we'll explore how STS validates tokens used with AssumeRoleWithWebIdentity, and using split-horizon DNS to facilitate AssumeRoleWithWebIdentity without exposing your infrastructure to the internet.
How STS validates a web identity token
When a workload calls
AssumeRoleWithWebIdentity , STS fetches the IdP's OIDC discovery document (.well-known/openid-configuration) and JSON Web Key Set (JWKS) to verify the token signature. STS makes these requests from the public internet and thus there has to be a public DNS record/public IP address and the ability to route directly to said IP without interference from a network appliance like a firewall. Workload AWS STS Issuer hostname | | | | 1. AssumeRoleWithWebIdentity(JWT) | |--------------------------->| | | | 2. GET {iss}/.well-known/ | | | openid-configuration | | |------------------------------>| | |<------------------------------| | | 3. GET {jwks_uri} | | |------------------------------>| | |<------------------------------| | | | | | 4. Verify signature, iss, | | | aud, exp, and role trust | | | policy conditions | | 5. Temporary credentials | | |<---------------------------| |STS AssumeRoleWithWebIdentity works like this at a high level:
- The
issclaim in the token matches the provider URL of an IAM OIDC identity provider in the account that owns the role.
- AWS STS uses public DNS to resolve the IP address(es) that belong with the domain used for your issuer. STS can only route to public IP addresses that are broadly accessible from the internet.
- The discovery document at
{issuer}/.well-known/openid-configurationis publicly reachable over HTTPS and containsissuerandjwks_uri.
- The
jwks_uriis publicly reachable over HTTPS, and contains at least one key
- The TLS certificate on the JWKS endpoint chains to a trusted public root CA. If it doesn't, IAM falls back to the thumbprints configured on the IAM OIDC provider.
- All relevant policies (Resource Control Policy, IAM Role Trust Policies, VPC endpoint policies) are considered, and if all conditions are met access is granted
How split-horizon DNS works
Split-horizon DNS is a concept where different DNS resolvers give different answers for a given domain. Using split horizon DNS means you give resolvers on your internal network a different answer for a name than the answer published on the public internet.
"Where is auth.example.com?" | +---------------------+---------------------+ | | Asked from inside Asked from the your network public internet | | v v +--------------------+ +--------------------+ | Internal resolver | | Public authoritative| | (internal view) | | DNS (public view) | +---------+----------+ +---------+----------+ | | v v 10.0.12.34 203.0.113.10 (full private IdP) (fully public IdP)- Internal view. On your private network, resolving the issuer/JWKS domains returns private IP addresses belonging to your infrastructure.
- Public view. On the internet, resolving the issuer/JWKS domains returns public IP addresses. These public IPs can be anything and are totally separate from your infrastructure. It could be content served through a load balancer, content delivery network, or anything can be given a certificate, and routable public IP.
Solution overview
Internal workloads, jobs, Actions AWS STS | | | resolves auth.example.com | resolves auth.example.com | (private location) | (public location) v v +----------------------------+ +------------------------------+ | Private OIDC System | | Public location | | - OIDC Discovery Doc | publish | Public OIDC discovery doc | | - JWKS endpoint IP |--------->| Public JWKS endpoint. | | | metadata | | | | | | +----------------------------+ +------------------------------+The design has three parts:
- The private OIDC system. This is your on-premesis/otherwise not internet accessible OIDC provider. This hosts your discovery doc, and possibly your JWKS endpoint. It has private IP address(es) or is by any means not accessible via the internet.
- The public location, reachable at the same hostname from the internet. it receives a copy of your discovery doc and JWKS. It has public IP address(es), and ideally a TLS certificate from a major trusted certificate authority.
- A publisher job that copies the current discovery doc and JWKS from the IdP to the public host whenever keys change.
For most customers I think an ideal public location is an s3 bucket fronted by cloudfront. This way AWS handles the certificate rotation, hosting the file, and there is minimal cost/overhead with no need to open anything in a firewall and all you need to handle is the DNS.
A solution using Cloudfront/S3 might look like this:
Private network | Public | +----------------------+ | | Private OIDC System | | | auth.example.com | | | (internal view) | | +----------+-----------+ | ^ | | 1. GET discovery doc + JWKS | | | +----------+-----------+ | +----------------------+ | Publisher | 2. PutObject | | Amazon S3 bucket | | Lambda in VPC, cron, |-------------------+--->| (private) | | or Autosys job | (outbound only) | +----------^-----------+ +----------------------+ | | | | origin access control | +----------+-----------+ | | Amazon CloudFront | | | auth.example.com | | | (public view) | | +----------+-----------+ | ^ | | 3. GET discovery doc + JWKS | | | +----------+-----------+ | | AWS STS | | +----------------------+
Checking Assumptions
This whole solution is caveated around one assumption, which is that the issuer hostname is in a domain you can publish publicly. STS resolves the issuer through public DNS. If your issuer uses an internal-only name, such as a
.internal or .corp suffix, you can't create a public view for it, and you would need to change the issuer URL first before following this pattern.The publisher job
The publisher job is the only new component you run. It can run on any scheduler that can reach your issuer and the public location:
Option | Use when |
|---|---|
AWS Lambda function in a VPC, invoked by Amazon EventBridge Scheduler | Your issuer is reachable from a VPC, and you don't want to manage a host. This would be my recommendation as it has credentials to write to S3 already, and requires very little overhead. It would need a route from your VPC, back to your private issuer. |
SSM run commands on EC2, or on-premises server with an SSM hybrid activation | This is a great option for getting data into S3, the run command can run with the instances credentials which can be granted access to write to a location in s3. SSM hybrid activations allows you to easily do this from an on-prem server without solving for AWS credential distribution first. |
cron on a host in your private network, or EC2 | Your issuer runs on premises, and you already have a host that can reach it. Running it on AWS compute like EC2 means you have already solved the write permissions to S3. If running outside AWS, you need to solve for the credential used to write to S3. |
Enterprise scheduler, such as Autosys | Your organization already runs and monitors jobs through a central scheduler. If running outside AWS, you need to solve for the credential. |
A step in your key rotation pipeline | Key rotation already runs through a pipeline. Keep a scheduled run as well, so the public copy stays in sync if a pipeline run fails. If running outside AWS, you need to solve for the credential used to write to S3. |
On each run, the job does the following:
- Fetches
{issuer}/.well-known/openid-configurationthrough the internal view.
- Confirms that the
issuervalue matches the issuer URL configured in IAM exactly, and that thejwks_uriis on the issuer hostname.
- Fetches the JWKS and confirms that it contains at least one key
- Writes the JWKS, then the discovery document, to the public location at the same paths they have on the issuer.
Rotate signing keys safely
Because STS reads the public copy of the JWKS, the public copy must contain a key before your issuer signs with it. Use this order:
Time --->
Key A [ signing ][ published, not signing ][ removed ] Key B [ published, not signing ][ signing ........... ^ ^ ^ | | | 1. Add key B to 2. Switch issuer 3. Remove key A after JWKS, wait for to sign with the longest token publish job and key B lifetime has passed caches- Add key B to your issuer's JWKS without signing with it. Wait for the publisher job's next successful run, plus the cache lifetime, plus time for STS to refresh its cached keys. Confirm that key B's
kidappears in the public JWKS.
- Switch your issuer to sign with key B.
- After the maximum token lifetime has passed, remove key A. The publisher job removes it from the public copy on its next run.
If you sign with a key before STS can see it,
AssumeRoleWithWebIdentity fails with InvalidIdentityToken .Validate the configuration
- From outside your network, the issuer hostname resolves to the public location, both documents are served over HTTPS, and other paths such as the token endpoint return an error.
- From inside your network, the issuer hostname resolves to your issuer's private address.
- The list of key IDs in the JWKS is the same inside and outside your network.
-
AssumeRoleWithWebIdentitysucceeds with a token from your issuer.
Security Considerations
- Any system that can write to the location of your JWKS or OIDC issuer may be able to gain access to your systems that trust your issuer. Consider publishing this data a sensitive activity.
- It is critical to ensure your public domain does not expire and fall under the possession of a bad actor, or else they may gain access to systems that trust your issuer
- Rotate keys in your JWKS in accordance with your security requirements
- Host the discovery doc/JWKS at a location that is resistant to denial of service attacks, or configurations that may be disrupted by network changes.
- If using AWS services like cloudfront/s3 to host your public version of your issuer, consider doing it in a dedicated AWS account in your organization that is separate from all business functions, and restricting access to that account to only authorized users.
Conclusion
Split-horizon DNS lets you keep your OIDC issuer on a private network while giving AWS STS the two public documents it needs to verify tokens. A scheduled publisher job copies those documents to a public location, so you keep your existing issuer URL, and open no inbound paths to your network.
Further reading
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article