AWS Builder Center

Builder Center

Nobody Follows the Onboarding Doc. So I Turned It Into a Portal Click.

Nobody Follows the Onboarding Doc. So I Turned It Into a Portal Click.

Building an Internal Developer Platform starter with Backstage, keyless Cosign signing, Argo CD, and Kyverno: five design decisions, five bugs, and an honest list of what it doesn’t do yet.

Otowo Samuel Odeh

An Internal Developer Platform starter built with Backstage, keyless Cosign signing, Argo CD, and Kyverno: five design decisions, five bugs, and an honest list of what it doesn't do yet.
Every platform team eventually writes the same document: create a repo, add a Dockerfile, wire up CI, add these labels… And every new service quietly skips half of it.
That's copy-paste drift, the tech debt no tracker shows. A team clones an old service to start a new one, inherits a deprecated CI action, keeps a Dockerfile from three Node versions ago, and drops the labels the policy engine requires. Six months later, nobody can say which services are compliant and which are archaeology.
More documentation won't fix that. A golden path will: a self-service template that generates repos which are correct by construction.
From three fields in a portal form, a developer gets:
  • A GitHub repo with CI that lints, tests, builds, and keyless-signs the container image
  • Kubernetes manifests that Argo CD syncs automatically
  • The labels the admission policy demands, already in place
  • OpenTelemetry and Prometheus hooks that work on first boot
  • A catalog entry with an owner, a system, and a lifecycle

How it works

1
2
3
4
5
6
graph LR
A[Developer] --> B[Backstage form]
B --> C[Template renders golden-path repo]
C --> D[CI signs the image]
D --> E[Argo CD syncs manifests]
E --> F[Catalog shows ownership]
The stack: Backstage (Software Templates v1beta3), GitHub Actions with Cosign keyless signing, Terraform modules for baseline AWS infrastructure, Argo CD for GitOps, Kyverno for admission policy, and OpenTelemetry plumbed in from line one.
I ran this end to end on AWS. The generated service answers on a public load balancer, Argo CD reports it Synced and Healthy, and the cluster runs the exact image CI built and signed. Terraform created the AWS side (an ECR repository, a CloudWatch log group, and a 5xx alarm) without anyone clicking through a console.

Five design decisions

1. The template generates a repo, not infrastructure

The scaffolder does three things: renders files, pushes a repo, registers the catalog entry. No terraform apply at click time, no cluster calls. The developer picks the golden path and fills in three fields: service name, description, owner. Ownership is rendered by the template and never hand-written, because an unowned service is an unmanaged service.
Why this works: every external call is a way for the template to fail for reasons unrelated to the developer's request. And the generated repo is inspectable: a reviewer opens GitHub and sees exactly what the platform approved.
The trade-off: provisioning happens on the first pipeline run instead of instantly. For a starter, a working, signed repo in under 15 minutes beats half-provisioned infrastructure.

2. Sign images keyless, or not at all

The number one reason teams skip image signing is key management. So the CI uses Cosign's keyless mode with GitHub OIDC: no key material anywhere in the repo, and verification is tied to the exact repository and workflow identity.
1
2
3
4
5
- run: cosign sign --yes "${{ steps.image.outputs.name }}@${{ steps.build.outputs.digest }}"
- run: |
cosign verify ... \
--certificate-identity-regexp "https://github.com/${GITHUB_REPOSITORY}/" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
All five Actions in the pipeline are SHA-pinned to their release commits. A pipeline that verifies everything else shouldn't float on mutable tags itself.
One honest note: the signature is verified inside the pipeline. Enforcing it at cluster admission is the next step (see What it doesn't do yet).

3. Tests that need zero cloud credentials

This is the decision I'd defend hardest. The Terraform modules use mock_provider "aws", and a Node render test scaffolds the template offline, then runs the generated repo's lint and unit tests against the output.
A project like this gets judged by whoever clones it, possibly with no AWS account, possibly on a plane. Tests that need credentials don't get run. Both suites finish in seconds, offline.

4. One policy, not a policy library

The Kyverno policy requires two labels (app.kubernetes.io/name and team) on every workload. Generated services carry them automatically, so the policy exists to stop hand-made services from reaching the cluster unowned.
One policy proves the compliance loop. Thirty policies become the project.
It's written as a Kubernetes-native CEL ValidatingPolicy (see bug #1 for why). The loop works end to end: a Deployment without the labels is denied at admission, and the same Deployment with the labels is admitted.

5. Observability that starts on first run

The OpenTelemetry SDK initializes only if OTEL_EXPORTER_OTLP_ENDPOINT is set. Otherwise it logs "tracing disabled" and the server runs. A generated service must start with zero external dependencies. Plugging in a real collector later means setting one environment variable, not rewriting the service.
The Terraform observability-baseline module provisions the CloudWatch side: a log group (/service/<name>/<env>) and a 5xx alarm on the load balancer, both tagged and created per service, ready before the first pod starts. The alarm watches HTTPCode_Target_5XX_Count, threshold 10, over two 5-minute periods. It's the cheapest possible safety net: no custom metrics, no agent, just the one number that means "something is broken."

Five bugs worth writing down

  1. Deprecated APIs don't announce themselves politely. I first wrote the admission policy as a Kyverno ClusterPolicy. Then I checked the docs properly: deprecated in v1.19 (August 2026), removed in v1.20. Shipping it would have meant a policy that silently stops working within one release cycle. The lesson: verify your "standard" patterns against the current release, not your memory.
  2. Invented config keys fail late. My Backstage guest-auth config used enabled: true and allowOutsideOfVisitor. Neither exists. The real API is guest: {}, with an ominously named dangerouslyAllowOutsideDevelopment escape hatch. Reading the provider's config.d.ts took 30 seconds and saved a broken startup.
  3. zsh doesn't word-split like bash. A loop fetching Action SHAs with set -- $spec passed the whole string as one argument:
    1
    fatal: unable to access 'https://github.com/actions/checkout v7.0.1.git/': URL rejected: Malformed input
    The fix, explicit parameter expansion (repo="${spec%% *}"), is elementary. The lesson is better: my test command and my shell disagreed, and the error pointed at git, not at me.
  4. The test claimed more than it ran. My README said npm test "renders template → lint → unit tests." It rendered. That's all. A claim in a README is a contract with whoever clones the repo, so I made the test actually run npm ci, lint, and unit tests on the rendered output.
  5. A lockfile with a template variable inside it. For npm ci to work in generated repos, the skeleton ships a package-lock.json whose name field contains ${{ values.service_name }}. Generating it meant swapping placeholders for literals, running npm install --package-lock-only, then swapping them back, and asserting after every render that both name fields resolved correctly.

Results

MetricResult
Portal click → green pipeline~2 minutes (30s scaffold + 1m 4s CI)
Generated services with keyless Cosign signing100% (it lives in the template)
Generated services with catalog ownership100% (rendered, never hand-written)
Reusable Terraform modules (tested, mocked)2, 8/8 tests passing
Credentials needed to run every test0
Template render test19 files, lint + unit tests green on output
The pipeline runs lint and unit tests first, then build, Cosign keyless sign, and verify, in 1m 4s end to end. The repo ships its own receipts: an architecture diagram, a decision log (decision, why, trade-off), a full build log of every command and failure, and a rebuild guide so someone can reconstruct the whole thing from an empty folder.

What it doesn't do yet

  • Signatures are verified in CI, not at the cluster. Kyverno enforces ownership labels but doesn't yet verify image signatures, so an unsigned image applied by hand could still be admitted. Next step: an image-verification policy pinned to the repo's workflow identity.
  • One environment. Only the dev overlay exists. Promotion path: add overlays/staging/ and overlays/prod/, register via ApplicationSet. Kustomize already supports it.
  • Runtime and cost. 2× t3.medium is about $0.22/hr (~$160/month). Scale to zero with eksctl scale nodegroup --nodes 0; the control plane stays at about $0.10/hr.
  • Backstage uses guest auth. Real SSO is out of scope for a starter.

Takeaways

  • Golden paths beat guidelines. A policy that has to be remembered will be forgotten; a template that's used can't be.
  • Keyless signing removed the excuse. Zero secrets, zero key management: the reason teams skip signing.
  • Offline tests are a feature of the demo, not just the code. Anyone who clones it gets green in under two minutes.
  • Currency is maintenance. Every pin (Actions, Node LTS, provider majors, policy APIs) was verified against October 2026 releases. A golden path with stale pins is self-defeating.
  • Write the README first, then make it true. Half my "bugs" were really gaps between what I claimed and what I ran.
If you run a platform team, what would you cut from this path, and what would you add?
Source, rebuild guide, and full build log: github.com/OtowoSamuel/idp-golden-path 
# kubernetes# devops# amazon-cloudwatch# amazon-eks# gitops
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article