kro v0.9 for Platform Engineering: Collections, External Refs, and Cluster Scope

Hands-on exploration of kro v0.9's headline features for platform teams: forEach collections, externalRef selectors with reactive watches, cluster-scoped instances, CEL enrichments (cel.bind, json.marshal, quantity), Graph Revisions, and a complete reproduce-it-yourself guide on kind.

allamand

Published Jun 17, 2026

Last modified Jun 17, 2026

This is a 2-part series:* Part 1 (this post): Collections, External Refs, Cluster Scope, CEL enrichments, Graph Revisions
Part 2 : omit() for field-level conditionals, Reconciliation limits, Enterprise readiness, Day 2 versioning

All RGDs, sample instances, and a cluster test plan are available in this GitHub Gist . See the TESTING.md file in the gist for validation steps on an EKS cluster.

TL;DR — Why this matters for platform teams

Platform teams building self-service APIs on EKS use kro (Kubernetes Resource Orchestrator) with ACK (AWS Controllers for Kubernetes) to expose opinionated ResourceGraphDefinitions (RGDs) to application teams.

Before v0.9, you had to deploy 5 separate instances to get 5 S3 buckets, and couldn't express "apply this policy to all gold-tier namespaces" without writing a custom controller. kro v0.9 solves both — one RGD, one instance, N reactive resources driven by cluster state.

Here's what v0.9 unlocks and how much work each adoption takes:

What's painful today	How v0.9 fixes it	Migration effort
Deploy N separate instances to get N S3 buckets	One `S3BucketSet` RGD with `forEach`	Write a new RGD
---	---	---
Deploy N instances for N pod-identity bindings	One RGD with `forEach` over bindings list	Write a new RGD
EKS/VPC instances pinned to an arbitrary namespace	Declare `scope: Cluster` in the schema	1-line change
IAM policies written as inline JSON strings	Build them with `json.marshal()` CEL	Rewrite CEL expressions
Argo Workflow starts before IAM policy is ready	Gate with resource-aware `includeWhen`	Add a condition
No way to fan out quotas across tenant namespaces	External ref selector + `forEach`	Write a new RGD
HPA/KEDA broken when `replicas` is in the RGD	Omit `replicas` from template — kro uses strategic merge, not replace	Remove the field

Maturity and risk

kro is v1alpha1. The API may introduce breaking changes across minor versions. The features in this post are validated on a kind cluster — not a recommendation to adopt blindly in production today.

Things to know before adopting:

RBAC: The default Helm install uses unrestricted mode. For production, switch to aggregation mode which requires a per-RGD ClusterRole.
EKS Capabilities: For production EKS clusters, kro is available as a managed add-on that handles RBAC and lifecycle. Note: you cannot choose the kro version — AWS controls the release cadence. To test v0.9 features today, use a self-managed install on kind or a dev cluster.

kro has ~2,900 GitHub stars and is a Kubernetes SIG Cloud Provider subproject with multi-vendor backing (AWS, Azure, GCP). It's lighter-weight than Crossplane for teams already on EKS + ACK.

The rest of this post is a hands-on exploration: adopt features incrementally, one RGD at a time, starting in dev/workshop environments.

Prerequisites

Docker >= 24
kind v0.24+
kubectl v1.36+
helm v3.20+ (optional, see install note below)

Installing kro v0.9

Option 1 — Helm (recommended):

1
2
3
4
5
6
`helm install kro oci://registry.k8s.io/kro/charts/kro \
  --version 0.9.2 \
  --namespace kro-system \
  --create-namespace \
  --set featureGates.CELOmitFunction=true
`

Option 2 — Raw manifests (if Helm is unavailable):

1
2
3
4
`kubectl create namespace kro-system
kubectl apply -f https://github.com/kubernetes-sigs/kro/releases/download/v0.9.2/kro-core-install-manifests.yaml
kubectl -n kro-system wait --for=condition=Available deploy/kro --timeout=120s
`

RBAC setup required. kro dynamically generates CRDs from your RGDs, and needs permission to manage them. With the Helm install above, this is handled automatically (unrestricted mode is the default). With raw manifests, the shipped ClusterRole is minimal — your RGDs will stay stuck in ControllerReady=False until you grant access. Quick fix for test/workshop clusters:

1
2
3
4
5
6
`# Test/workshop only — grants kro full cluster access
kubectl create clusterrolebinding kro-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kro-system:kro
kubectl -n kro-system rollout restart deploy/kro
`

⚠️ Production: Use Helm with aggregation mode (--set rbac.mode=aggregation) and create a per-RGD ClusterRole for each resource type. See the access control docs for examples.

Sharp edges found during testing

These are the gotchas I hit while validating the examples on a kind cluster. Each is explained in detail where it first appears in the post.

What happens	When you'll hit it	Fix
RGDs stuck in `ControllerReady=False`	Raw-manifest install (Helm `unrestricted` mode handles this)	Use Helm, bind kro SA to `cluster-admin` for testing, or provide dedicated per-RGD permissions for production
---	---	---
`status` expression fails with "unknown identifiers: [schema]"	You reference `schema.spec.*` in the instance `status` block	Derive status values from resource outputs instead
`map.merge()` not available	You want to write `labels: ${map.merge(platformLabels, schema.spec.extraLabels)}` to let users add custom labels alongside your platform defaults. Despite being listed in kro's CEL libraries docs , the `map` identifier isn't recognized in v0.9.2. This may ship in a future release.	Define all labels statically in the template, or let users pass labels via a Kyverno/webhook policy applied after kro creates the resource
`IAMRoleSelector` rejected at RGD creation	Your RGD sets `metadata.namespace` on `IAMRoleSelector` (which is cluster-scoped). Previous versions silently ignored this; v0.9 now validates resource scope at compile time and rejects the mismatch.	Remove `metadata.namespace` from the template — this is a one-line fix per RGD
kro resets `replicas` set by HPA/KEDA	Your RGD template includes `replicas: ${schema.spec.replicas}` and an HPA or KEDA ScaledObject targets the same Deployment	Remove `replicas` from the RGD template entirely — let HPA/KEDA own it. See the Reconciliation limits section

Tip: Upgrade kro first, then re-apply your RGDs. Existing RGDs continue to reconcile under the old Graph Revision until you re-apply them.

1. Collections (`forEach`)

Platform teams often need to create multiple identical resources from a single request — five S3 buckets for a data pipeline, pod-identity bindings for every microservice, or a Deployment per region. Without collections, you either duplicate instances in Git (one per resource) or write a custom controller. forEach lets a single RGD instance produce N resources from a list, keeping your Git repo clean and your intent explicit.

forEach has existed for a few minor releases, but v0.9 is where it becomes safe and productive. The feature is now hard-capped at 1000 elements and 10 dimensions (configurable via --rgd-max-collection-size and --rgd-max-collection-dimension-size) for cluster-safety guardrails (#1036 , #1045 ). Collection watches are registered for every expanded resource (#1182 ), so per-element status propagation is reliable. v0.9.2 added duplicate-name validation (#1244 ) so that a forEach producing two items with the same resolved metadata.name fails fast instead of creating conflicting writes. It also fixed a resource leak when a collection ref points to an empty set (#1249 ).

What it looked like before

A typical S3Bucket RGD provisions one bucket per instance. To get five buckets, you deploy five instances via Argo CD, which clutters your Git repo and dilutes intent.

With v0.9 Collections

1
2
3
4
`apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
# ... (full manifest in companion gist)

📎 View complete manifest in the companion gist

One instance produces N buckets, tracked as a set, reconciled in one graph. If a data team later adds "inference-logs" to the list, kro creates the new bucket, leaves the others alone, and the whole thing lives under a single Argo CD Application.

Design consideration: forEach works best when all items share the same configuration. If bucket configurations diverge later (different lifecycle policies, different regions), you'll need to refactor back to individual instances or use a richer schema with per-bucket config objects.

Note: The YAML comment above (# IAMRoleSelector is cluster-scoped) reflects the fix — see the Sharp edges table for details.

Cartesian product: multi-region x multi-tier workloads

forEach supports multiple dimensions. When you specify two lists, kro generates one resource for every combination — a cartesian product. This is useful when you need the same workload deployed across multiple regions and tiers without duplicating YAML:

1
2
3
4
5
6
7
8
9
10
`- id: deployments
  forEach:
    - region: ${schema.spec.regions}    # ["us-east-1","eu-west-1"]
    - tier:   ${schema.spec.tiers}      # ["web","api"]
  template:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ${schema.metadata.name + '-' + region + '-' + tier}
`

Four Deployments from two short lists. Be mindful of dimension explosion: 3×5×10 = 150 resources from a single instance. The 1000-element cap protects you, but plan your cardinality.

v0.9.2 also documents a map().flatten() pattern for cartesian products across collections without exceeding the dimension guard:

1
2
3
4
5
`- id: pairs
  forEach:
    items: ${schema.spec.buckets.map(b, schema.spec.regions.map(r, {"bucket": b, "region": r})).flatten()}
  template: { ... }
`

Known limitation: Collections exceeding the 1000-resource limit can get stuck during deletion, requiring manual cleanup. Size your collections conservatively.

Collections from the cluster: external refs with selectors

Sometimes you don't know the list upfront — it comes from the cluster itself. A platform team wants to say "apply a quota to every namespace labelled gold" or "create a network policy for every namespace owned by this tenant." The resources to act on are dynamic: teams create namespaces, label them, and expect the platform to react. Before v0.9, this required a custom controller watching namespaces and reconciling child resources. Now, externalRef with a label selector discovers matching resources dynamically, and kro watches them reactively — when a new match appears, dependent resources re-reconcile within seconds.

externalRef used to require a specific metadata.name. It now accepts a label selector and exposes a full array of matches:

1
2
3
4
5
6
7
8
9
10
11
`- id: tenantNamespaces
  externalRef:
    apiVersion: v1
    kind: Namespace
    metadata:
      selector:
        matchLabels:
          peeks.aws/tenant-tier: gold
        matchExpressions:
          - { key: peeks.aws/decommissioned, operator: DoesNotExist }
`

tenantNamespaces is now an array that kro watches reactively: when a namespace is labelled tenant-tier=gold, every dependent instance re-reconciles within seconds (#1079 , #1180 ). Combined with forEach, this unlocks fleet patterns that previously required a custom controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
`- id: quotas
  forEach:
    - ns: ${tenantNamespaces}
  template:
    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: gold-quota
      namespace: ${ns.metadata.name}
    spec:
      hard:
        pods: "200"
        requests.cpu: "100"
`

A single GoldTenantPolicy instance blankets every gold-labelled tenant namespace with a quota, and new namespaces are picked up automatically. On a lightly-loaded kind cluster, the reactive watch triggered within two seconds of labelling a namespace.

Blast radius note: If namespaces are labelled gold broadly, quotas fan out to all of them. The 1000-element cap on forEach provides a hard ceiling, but use tight matchExpressions (e.g., require both tenant-tier=gold AND a specific owner label) to limit the scope intentionally.

2. Cluster-scoped instances (KREP-10)

Some platform resources don't belong to any namespace — an EKS cluster, a VPC, or a "tenant" that creates namespaces. These are cluster-wide concerns owned by the platform team, not by application teams in their namespaces.

Platform primitives split naturally into two groups:

Namespaced: S3Bucket, PodIdentity, HuggingFaceModel, AppmodService
Cluster-wide: EKSCluster, VPC, RayService, and anything "tenant factory"-shaped

Before v0.9, all instances were namespaced, which forced awkward patterns like pinning EKS/VPC instances to an arbitrary kro-system namespace. Now you declare scope: Cluster in the schema:

1
2
3
4
5
6
7
8
9
10
`spec:
  schema:
    apiVersion: v1alpha1
    kind: EKSCluster
    scope: Cluster           # NEW in v0.9
    spec:
      name: string | required=true
      vpcId: string | required=true
      targetNamespace: string | default="kube-system"
`

Child namespaced resources (ACK Cluster, Nodegroup, addons) must then set metadata.namespace explicitly. kro validates this at RGD creation, which acts as a forcing function for clarity.

This also unlocks a clean Tenant factory pattern:

1
2
3
4
`schema:
  kind: Tenant
  scope: Cluster
# ... (full manifest in companion gist)

📎 View complete manifest in the companion gist

3. `includeWhen` references other resources

In multi-step provisioning, resources have dependencies: you can't start a model download workflow until the IAM policy exists and the pod identity is bound. Without conditional logic, you end up with stuck workflows retrying against resources that don't exist yet, or you split the RGD into multiple stages applied manually in sequence.

v0.9 lets includeWhen conditions depend on other resources in the graph, with proper DAG tracking and cycle detection. kro only creates the gated resource once upstream conditions are met.

A concrete example: you only want to create an Argo Workflow to download a model after the S3 policy has an ARN and the Pod Identity is bound.

1
2
3
4
5
6
7
8
9
`- id: downloadWorkflow
  includeWhen:
    - ${s3Policy.status.conditions.exists(c, c.type == "ACK.ResourceSynced" && c.status == "True")}
    - ${piAssociation.status.conditions[0].status == "True"}
  template:
    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    # ...
`

No more "pending policy ARN" stuck workflows. The expression participates in the DAG, so kro re-evaluates it on upstream changes.

4. CEL library enrichments

RGD authors frequently hit CEL limitations: repeating long expressions, building JSON IAM policies as escaped strings, or comparing Kubernetes quantities like "8 CPU". v0.9 adds nine CEL capabilities that eliminate these workarounds. The most useful ones:

1
2
3
`# CEL helpers used in this RGD: size(), json.marshal(), cel.bind()
# Reference: https://kro.run/docs/concepts/cel/cel-libraries
`

cel.bind() — assign a subexpression to a variable so you don't repeat it. Syntax: cel.bind(name, value, expression_using_name):

1
2
3
4
5
6
7
8
`# Without cel.bind() — the prefix is computed twice:
name:   ${schema.metadata.name + '-' + region + '-web'}
suffix: ${schema.metadata.name + '-' + region + '-svc'}

# With cel.bind() — compute the prefix once as 'base', then reuse it:
name: ${cel.bind(base, schema.metadata.name + '-' + region, base + '-web')}
# Reads as: let base = "myapp-us-east-1", then return "myapp-us-east-1-web"
`

In practice, this is useful when multiple fields in a template share a computed prefix:

1
2
3
4
5
6
7
8
9
10
11
`- id: tenantRole
  template:
    apiVersion: iam.services.k8s.aws/v1alpha1
    kind: Role
    metadata:
      name: ${cel.bind(prefix, "peeks-" + schema.spec.name, prefix + "-role")}
      # prefix = "peeks-team-ml" → name = "peeks-team-ml-role"
    spec:
      name: ${cel.bind(prefix, "peeks-" + schema.spec.name, prefix + "-role")}
      description: ${"IAM role for tenant " + schema.spec.name}
`

quantity — compare Kubernetes resource values natively:

1
2
3
`readyWhen:
  - ${quantity(deployment.status.?allocatable.cpu).compareTo(quantity("8")) >= 0}
`

json.marshal / json.unmarshal — generate IAM policy documents without multi-line heredocs. Instead of embedding a large inline JSON policy as a string literal, v0.9 lets you build it structurally:

1
2
3
4
5
6
7
8
9
`policyDocument: ${json.marshal({
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject","s3:PutObject"],
    "Resource": ["arn:aws:s3:::" + schema.spec.s3Bucket + "/*"]
  }]
})}
`

omit() (feature-gated: CELOmitFunction) — drop fields entirely when empty, which matters for ACK resources that distinguish "absent" from "empty":

1
2
`kmsKeyID: ${schema.spec.kmsKeyID != "" ? schema.spec.kmsKeyID : omit()}
`

map.merge() — merge user labels with platform-standard ones:

1
2
3
4
5
`metadata:
  labels: ${map.merge({"peeks.aws/managed-by": "kro",
                      "peeks.aws/tier": schema.spec.tier},
                      schema.spec.extraLabels)}
`

Known issue: map.merge() does not resolve by default. In testing, this pattern failed with references unknown identifiers: [map]. This appears to be behind an undocumented feature gate. If you need label merging today, fall back to statically-declared labels and push merging into a downstream controller (Kyverno, webhook).

Known issue: schema.spec.* not visible in status expressions. Inside resource templates, schema.spec.foo works. Inside the instance status block, the same expression fails with references unknown identifiers: [schema]. Status expressions in v0.9 see resource names only:

1
2
3
4
5
6
7
8
9
`# WRONG — fails to compile
status:
  bucketCount: ${string(size(schema.spec.buckets))}

# RIGHT — status traverses resources
status:
  allReady: ${buckets.all(b, b.status.conditions.exists(
    c, c.type == "ACK.ResourceSynced" && c.status == "True"))}
`

5. Graph Revisions (KREP-013)

When a platform team pushes a broken RGD update, every instance using that RGD breaks — and the error surfaces on each instance individually, making it hard to tell whether the problem is the RGD or the instance. Platform teams need a way to validate RGD changes before they affect consumers, and to roll back if something goes wrong.

Every change to an RGD spec now creates an immutable GraphRevision:

1
2
3
`kubectl get gr            # short name for GraphRevision
kubectl describe gr <name>
`

Compilation errors surface on the revision, not on every orphaned instance. kro keeps the last 5 revisions (configurable) and this groundwork will enable revision pinning and progressive rollouts in future releases. For a platform team owning shared RGDs, this is the first step toward a proper "RGD release process" where you review a revision before consumers roll forward.

kubectl describe rgd s3bucket.kro.run now tells you exactly which revision is active and which failed. v0.9.2 added constraint-change detection (#1276 ): tightening a quota bound (maximum: 1000 -> 500) on an existing CRD now fails the Graph Revision check instead of silently invalidating stored instances.

How this interacts with ArgoCD: Graph Revisions track kro-internal compilation state. ArgoCD tracks Git-level changes. They're complementary — ArgoCD tells you what changed in Git, Graph Revisions tell you whether kro accepted it. A failed Graph Revision will show as a degraded RGD in your ArgoCD dashboard.

6. Operational niceties

Beyond the headline features, v0.9 includes smaller improvements that matter when you run kro on shared clusters with real workloads — break-glass controls, observability, safer upgrades, and better Argo CD integration:

Disable reconciliation per instance with annotation kro.run/reconcile: disabled (#1191 ). Handy for break-glass operations on a production bucket.
Controller metrics (pkg/runtime, RGD and instance controllers, CEL cache hit/miss). This gives kro a Prometheus story for the first time.
Helm chart exposure: topologySpreadConstraints, leaderElection tuning, feature gates, pprof build tag.
Schema comparison: changing a field's default= is now correctly classified as non-breaking.
Applyset orphan pruning respects reverse topological order (#1210 ). When a Tenant is deleted, child resources are torn down leaf-first. No more ACK controllers stuck on "dependency still exists".
listType and listMapKey markers in SimpleSchema (#1274 ). Declare buckets as a keyed list (by name) so Argo CD diffs stay stable instead of re-ordering the list on every sync.
Float support (#1255 , #1265 ). RGDs exposing resource-quota values or CPU/memory fractions via spec.*: float now survive the CEL type-check.
Default-value type validation (#1270 ). Default values that don't match the declared type are rejected at RGD install instead of failing at instance creation.
CRD conflict protection (#1241 ). A CRD name collision on upgrade no longer takes out children of unrelated RGDs.

Note: Instance condition events were added in v0.9.0 (#1208 ) then reverted in v0.9.2 (#1280 ). Do not enable the InstanceConditionEvents feature gate until v0.9.3+.

Reproduce it yourself on `kind`

Everything below was executed on a fresh Linux VM (arm64, Docker 25, kind v0.24, kubectl v1.36). Copy-paste friendly.

1. Clone the companion files

1
2
3
`git clone https://gist.github.com/82fb7ed6c91c1fccf227b07185d7d600.git /tmp/kro-v09-peeks
cd /tmp/kro-v09-peeks
`

2. Cluster + kro install

1
2
3
4
`kind create cluster --name kro-test --wait 180s

kubectl create namespace kro-system
# ... (full manifest in companion gist)

📎 View complete manifest in the companion gist

3. Install ACK CRDs (no controllers needed for compile validation)

1
2
3
4
`# Pin to specific tags for reproducibility
for u in \
  https://raw.githubusercontent.com/aws-controllers-k8s/s3-controller/v1.0.17/config/crd/bases/s3.services.k8s.aws_buckets.yaml \
# ... (full manifest in companion gist)

📎 View complete manifest in the companion gist

4. Apply the RGDs

1
2
3
4
5
6
7
`kubectl apply -f rg-gold-tenant-policy.yaml \
              -f rg-s3bucket-set.yaml \
              -f rg-tenant.yaml
sleep 15
kubectl get rgd
# All three: STATE=Active, READY=True
`

5. Exercise forEach (S3 buckets)

1
2
3
4
5
`kubectl apply -f instance-s3bucket-set.yaml
sleep 10
kubectl get buckets.s3.services.k8s.aws
# peeks-demo-raw / peeks-demo-curated / peeks-demo-models
`

6. Exercise cluster-scoped Tenant + reactive GoldTenantPolicy

1
2
3
4
`kubectl apply -f instance-tenant.yaml \
              -f instance-gold-tenant-policy.yaml
sleep 10
# ... (full manifest in companion gist)

📎 View complete manifest in the companion gist

7. Verify Graph Revisions

1
2
3
`kubectl get gr
kubectl describe rgd s3bucketset.kro.run | grep -A5 "Active Revision"
`

8. Cleanup

1
2
`kind delete cluster --name kro-test
`

Reproducing known issues: See TESTING.md in the companion gist for step-by-step reproduction of the status scope limitation, map.merge() failure, and IAMRoleSelector namespace rejection.

Next up: In Part 2 , we explore omit() for field-level conditionals, reconciliation limits with HPA/KEDA/Argo Rollouts, and enterprise readiness (kro vs KubeVela gap analysis).

# containers # gitops # amazon-eks

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

kro v0.9 for Platform Engineering: Collections, External Refs, and Cluster Scope

allamand

TL;DR — Why this matters for platform teams

Maturity and risk

Prerequisites

Installing kro v0.9

Sharp edges found during testing

1. Collections (forEach)

What it looked like before

With v0.9 Collections

Cartesian product: multi-region x multi-tier workloads

Collections from the cluster: external refs with selectors

2. Cluster-scoped instances (KREP-10)

3. includeWhen references other resources

4. CEL library enrichments

5. Graph Revisions (KREP-013)

6. Operational niceties

Reproduce it yourself on kind

1. Clone the companion files

2. Cluster + kro install

3. Install ACK CRDs (no controllers needed for compile validation)

4. Apply the RGDs

5. Exercise forEach (S3 buckets)

6. Exercise cluster-scoped Tenant + reactive GoldTenantPolicy

7. Verify Graph Revisions

8. Cleanup

1. Collections (`forEach`)

3. `includeWhen` references other resources

Reproduce it yourself on `kind`