AWS Builder Center
kro v0.9 Deep Dive: omit(), Reconciliation Limits, and Enterprise Readiness

kro v0.9 Deep Dive: omit(), Reconciliation Limits, and Enterprise Readiness

Part 2 of our kro v0.9 exploration: the omit() function for field-level conditionals without duplication, reconciliation behavior with HPA/KEDA/Argo Rollouts (validated compatibility matrix), enterprise gap analysis vs KubeVela, and Day 2 RGD versioning strategies for fleet-scale deployments.

> This is Part 2 of our kro v0.9 exploration. Part 1  covers forEach collections, externalRef selectors, cluster-scoped instances, CEL enrichments, and Graph Revisions.

The omit() function: solving composition without duplication

kro v0.9.0 introduced a feature-gated CEL function called omit() that solves one of the most painful composition problems: field-level conditional inclusion.

The problem

In complex applications like in the appmod-blueprints  reference architecture, where the appmod-service.yaml RGD needs to support multiple optional capabilities: ingress, blue/green rollouts, HPA autoscaling, IAM role annotations, etc. Without field-level conditionals, you face a combinatorial explosion: with N optional features, you need up to 2^N near-identical resource blocks using includeWhen at the resource level. Any structural change must be replicated across every variant.

The solution: omit()

Enable the feature gate:
1
2
3
4
5
`# Helm values
config:
featureGates:
CELOmitFunction: true
`
Or pass the flag directly:
1
2
`--feature-gates=CELOmitFunction=true
`
Then use omit() in CEL expressions to conditionally remove a field entirely:
1
2
3
4
`apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
# ... (full manifest in companion gist)

Why this matters: Server-Side Apply field ownership

When omit() drops a field, kro does not take ownership of that field via Server-Side Apply. We validated this on a kind cluster:
1
2
3
4
5
6
7
8
`# With omit() → kro does NOT own replicas
Manager: kro.run/applyset
→ replicas NOT managed ✅

# Without omit() → kro DOES own replicas
Manager: kro.run/applyset
→ replicas IS managed ❌ (reset on every reconcile)
`
This means: - A field removed by omit() can be freely managed by HPA, KEDA, Argo Rollouts, or any external controller - kro will never reset it back, because it never claimed ownership - The field is still created with K8s API defaults (e.g., replicas defaults to 1)

Combining omit() with includeWhen

The combination of both mechanisms gives you full composition control in a single RGD — no duplication:
MechanismScopeUse case
includeWhenEntire resourceOptional Ingress, HPA, NetworkPolicy
---------
omit()Single fieldOptional annotation, replicas delegation, conditional selector
A real-world example — one RGD, four deployment profiles:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
`resources:
- id: deployment
template:
# ...
spec:
replicas: "${schema.spec.replicas \> 0 ? schema.spec.replicas : omit()}"
# ...
- id: ingress
includeWhen:
- "${schema.spec.enableIngress}"
template:
# ... only created when ingress is enabled
- id: hpa
includeWhen:
- "${schema.spec.enableHPA}"
template:
# ... only created when HPA is enabled
`
Results from our kind cluster tests:
Instancereplicas set?IngressHPAActual replicas
minimal (replicas=0, no ingress, no HPA)omitted❌❌1 (K8s default)
---------------
full (replicas=0, ingress=true, HPA=true)omitted✅✅ min=22 (HPA min)
hpa-only (replicas=0, HPA=true)omitted❌✅ min=33 (HPA min)
fixed (replicas=5)5❌❌5 (kro-managed)

Solving the reconciliation problem with omit()

Remember the reconciliation limits? The previous workaround was to simply not put the field in the RGD template at all. But what if you want the field to be configurable — sometimes set by the user, sometimes delegated to an external controller?
omit() bridges this gap:
1
2
3
4
`# User sets replicas=3 → kro manages it (field IS in the SSA payload)
# User sets replicas=0 → kro omits it (field NOT in the SSA payload → HPA free to scale)
replicas: "${schema.spec.replicas \> 0 ? schema.spec.replicas : omit()}"
`
This is the missing piece between "always managed by kro" and "never in the template". It gives the instance author the choice, per-deployment.

Limitations and caveats

  1. Alpha feature gate — CELOmitFunction must be explicitly enabled. Not available on EKS Capabilities (managed kro) yet.
  2. Scalar fields only — omit() removes the containing field. You cannot conditionally omit a nested block (e.g., an entire strategy: section). For that, use includeWhen on a separate resource.
  3. Required metadata fields — omit() is rejected on metadata.name, metadata.namespace, apiVersion, kind.
  4. CEL type safety — The return type of the ternary must match the field type. omit() is polymorphic but the "truthy" branch must be the correct type.
Reproduce it yourself: Enable the feature gate, apply the manifests from the companion gist  (rg-webapp-omit.yaml), and verify with kubectl get deploy -o jsonpath='{.metadata.managedFields}'.

What's next for kro

The v0.10 milestone  has 8 KREPs scheduled. The ones most relevant to platform teams:
  • KREP-006 — Propagation Control (propagateWhen): Gate mutations as changes propagate through the graph. Complementary to readyWhen — together they bookend when a resource can start mutating and when it's considered complete. This will give finer control over rollout ordering in multi-resource RGDs.
  • KREP-011 — RGD Variables (data resources): Define computed data inside an RGD that isn't a Kubernetes resource — think shared constants, computed ARNs, or intermediate values reusable across templates without creating a ConfigMap.
  • KREP-021 — kro CLI: A dedicated CLI for validating, testing, and managing RGDs locally before applying them to a cluster.
  • KREP-014 — Resource Lifecycles: Control what happens when resources are removed from an RGD — delete, orphan, or retain.
  • KREP-005 — Level-based Topological Sorting: Smarter parallelism in resource creation based on graph depth levels.
  • Detach on deletion (#183 , lifecycle/frozen): Delete an RGD instance while keeping its child resources alive — proposed annotation kro.run/detach-on-deletion: "true". Not yet implemented. Note: kro.run/reconcile: disabled (#1191 ) pauses reconciliation but does NOT prevent cascade deletion — the finalizer still fires on kubectl delete.
Further out, v1.0 is on the roadmap but has no KREPs assigned yet — expect API stabilization and graduation from v1alpha1 to happen there.
Follow progress on the kro roadmap page  or the #kro Slack channel .

Reconciliation limits: what kro resets and what it leaves alone

kro reconciles resources using a strategic merge patch — which means it only touches fields that appear in the RGD template. This has profound implications when kro-managed resources interact with other controllers (HPA, KEDA, Argo Rollouts, sidecar injectors).
We validated these behaviors on a kind cluster with kro v0.9.2 and filed them upstream as kubernetes-sigs/kro#1290 .
Reproduce it yourself: The companion gist  contains all manifests (rg-webapp.yaml, rg-webapp-hpa-compatible.yaml, webhook injector) and a step-by-step test plan (TESTING-RECONCILIATION.md) you can run on a local kind cluster in ~5 minutes.

What kro resets (❌ conflicts)

ControllerField affectedBehavior
HPAdeployment.spec.replicaskro resets to RGD value in <2s
---------
KEDA ScaledObjectdeployment.spec.replicas (via KEDA-managed HPA)kro resets in <3s
Argo Rolloutsservice.spec.selector (adds hash label)kro removes the hash immediately
Any controller modifying a field present in the RGD templateThat fieldReset on every reconcile loop (~1-3s)
The root cause: kro treats every field in the RGD template as authoritative. If replicas: 2 is in your RGD, kro will always set it back to 2, regardless of what the HPA decides.

What kro leaves alone (✅ coexistence)

MutationWhere it happensBehavior
Pod-level sidecar injection (Istio, Linkerd, custom webhooks)MutatingWebhook on Pod CREATE✅ kro manages the Deployment template; the webhook mutates Pods at creation — no conflict
---------
initContainers added to Deployment templateWebhook or external controller patches the Deployment✅ Persists indefinitely — kro doesn't specify initContainers in the template, so it ignores them
Extra containers added to Deployment templateSame pattern (Vault Agent, envoy sidecar)✅ Persists — fields absent from the RGD template are not reconciled
Annotations/labels added by external systemsAny controller or webhook✅ Persists — as long as they're not in the RGD template
The key insight: kro's merge behavior is additive for fields it doesn't own, and authoritative for fields it does. If you don't put it in the template, kro won't touch it.

Workarounds for existing RGDs

Problem: You want HPA/KEDA autoscaling on a Deployment managed by kro.
Solution: Don't put replicas in the RGD template. Let it default via the Kubernetes API server (which defaults to 1), then let HPA/KEDA manage it:
1
2
3
4
`# ❌ BAD — kro will fight the HPA
resources:
- id: deployment
# ... (full manifest in companion gist)
Problem: You use Argo Rollouts which adds a -hash label to Service selectors during canary/blue-green.
Solution (validated ✅): Don't specify the selector in the Service template. Argo Rollouts will inject its own rollouts-pod-template-hash selector at runtime, and kro will leave it alone since it's not in the RGD. We tested a full blue/green cycle (deploy → preview → promote → abort/rollback) with kro v0.9.2 and Argo Rollouts v1.7.2 — zero conflicts.
Problem: You want Vault Agent or Istio to inject sidecars.
Solution: No workaround needed! Pod-level webhooks coexist naturally. Even Deployment-level mutations (initContainers, extra containers) persist as long as they add fields not present in the RGD template.

Compatibility matrix

PatternCompatible with kro?Condition
HPA / KEDA / VPA✅ YesRemove replicas from RGD template
---------
Argo Rollouts (blue/green)✅ YesDon't specify Service selector in RGD — Rollouts injects its hash-based selector at runtime
Istio / Linkerd sidecar✅ YesNo changes needed
Vault Agent Injector✅ YesNo changes needed
Kyverno mutating policies✅ YesAs long as they add/modify fields not in the RGD template
Custom admission webhooks✅ YesSame rule — don't mutate fields kro owns
Future fix: The kro team is tracking ignoreFields / managementPolicy proposals that would let you explicitly delegate ownership of specific fields to other controllers. Until then, the workaround is structural: keep runtime-mutable fields out of your RGD templates.

Composition & Ordering

kro v0.9's graph-based reconciliation unlocks two composition features that matter for platform teams: ordered resource creation and collection iteration.

readyWhen — Execution ordering via dependency gates

kro has no explicit dependsOn field. Instead, ordering is inferred from CEL references between resources. If resource B references ${a.metadata.name}, kro places B after A in the topological sort. Combined with readyWhen, this creates proper deployment sequencing:
1
2
3
4
`resources:
- id: config
template:
# ... (full manifest in companion gist)
Observed behavior (validated on kind): - ConfigMap + Deployment created at T+0 (ConfigMap's readyWhen is instantly satisfied) - Service created at T+7s — only after Deployment reached readyReplicas \>= 1 - Topological order: ["config", "deployment", "service"]
Key insight: The readyWhen gate only delays downstream creation when the expression takes time to become true. If your upstream readyWhen is instantly satisfied (e.g., checking name != ''), kro creates both in the same reconcile cycle.
Syntax pitfalls (discovered via testing): - readyWhen can only reference the resource's own status fields — not schema.* - Status fields in the RGD schema must be CEL expressions referencing resources, not bare type declarations - Use string() for type casting when injecting numeric/int values into string annotations

forEach — Collection iteration with custom types

forEach expands a single resource definition into N instances from a list. The syntax requires custom types for object arrays:
1
2
3
4
`spec:
schema:
apiVersion: v1alpha1
# ... (full manifest in companion gist)
Edge cases validated (all passed ✅ on kind v0.9.2):
CaseResult
Empty array ([])Instance ACTIVE immediately, 0 resources
------
Single item1 namespace + 1 deployment
20 items20 NS + 20 deploys in ~30s
Add item dynamicallyResources created
Remove items (4→2)Orphaned resources deleted cleanly
Modify replicas in-placeUpdated without recreating
Syntax pitfalls: - forEach must be an array of {varName: "${CEL}"} objects (not a bare CEL string) - Array of objects in the schema requires a types: section — inline []{} notation is not supported - Cartesian product (multiple forEach dimensions) is supported but untested

readyWhen + forEach + omit() + includeWhen = Full composition toolkit

With v0.9.2, kro now covers four orthogonal composition axes:
AxisMechanismScope
Resource-level conditionalincludeWhenInclude/exclude entire resources
---------
Field-level conditionalomit() (CELOmitFunction)Include/exclude individual fields
IterationforEach + custom typesN instances from a list
OrderingCEL ref graph + readyWhenSequential creation with health gates
This means a single RGD can express: "For each team in the list, create a namespace, optionally create an HPA only if enableHPA is true, omit the replicas field when HPA is active, and don't create the Ingress until the Service is ready."

Enterprise gaps — kro vs KubeVela requirements

After extensive testing of kro v0.9.2 against OAM/KubeVela patterns , here's a gap analysis for enterprise platform teams. Each requirement is mapped to existing kro solutions or upstream proposals.
#RequirementKubeVela equivalentkro v0.9 statusUpstream issue/KREP
1Modular composition — import RGD modules, compose RGDs from smaller RGDsComponents + Application❌ Not supported — nested RGDs exist but have reconciliation loop bugs #1293  (deferred schema), #1057 
---------------
2Declarative permission/policy — conditional IAM annotations, policy injectionTraits (labels/annotations trait)✅ Solved — omit() for conditional annotations (iamRoleArn != '' ? val : omit())#1290 
3Apply-once / field management — don't fight external controllersTrait takeover mode✅ Solved — omit() removes SSA ownership; external controllers manage freely#1290 
4Custom health checks & ordering — wait for cert-manager, operator CRD readinessHealth probes in workflow⚠️ Partial — readyWhen works but limited to resource's own fields. No HTTP probes, no shell exec#542  (lifecycle modifiers)
5Global application context — cluster name, region, account ID accessible to all RGDsApplication context⚠️ Partial — externalRef can read ConfigMaps/Secrets. No built-in global context injection#920  (Global Variables, needs-krep)
6Iteration/looping — for each tenant, for each regionComponentDefinition with for✅ Solved — forEach with custom types, dynamic add/remove, 0-N items, cartesian productAlready in v0.9
7Traits system — cross-cutting concerns (sidecar injection, observability, network policies)Traits (TraitDefinition)❌ Not supported — no mechanism to apply cross-cutting mutations across multiple RGDs#1048  (Extension mechanism, stale)
8Controller stability — no reconciliation fights, predictable behavior under loadN/A (stable since v1.0)⚠️ Partial — SSA-based, predictable for managed fields. But any field in template gets force-reconciled (known)—
9Field-level conditionals — without duplicating entire resourcesPatch trait with conditions✅ Solved — omit() on scalars + includeWhen on resources. Alpha, scalar-only limitation#928 

Summary: 4 solved, 3 partial, 2 gaps

Solved (4/9): Field management (#2, #3), iteration (#6), field-level conditionals (#9)
Partial (3/9): Health checks (#4 — readyWhen covers 80% of cases), global context (#5 — externalRef workaround), stability (#8 — predictable but opinionated)
Gaps (2/9): Modular composition (#1), traits (#7)
The two remaining gaps (#1 modular composition, #7 traits) are architectural — they require new abstractions at the kro controller level. For platform teams today: - #1 workaround: One "fat" RGD per team type (vs. composing small reusable blocks) - #7 workaround: Kyverno/Gatekeeper policies + admission webhooks for cross-cutting concerns

Upcoming KREPs addressing gaps (v0.10 roadmap)

KREPTitleAddresses
KREP-006 Propagation Control (propagateWhen)#4 partially — gates when mutations propagate
---------
KREP-011 RGD Variables (data resources)#5 — complex data without K8s resources
KREP-014 Branch-aware dependency inference#4 — smarter ordering with conditional branches
#920 Global Variables (needs-krep)#5 — cluster-level context injection
#892 RBAC isolation for RGDs (needs-krep)#2 — permission boundaries

Day 2: RGD versioning & progressive migration

The biggest production question isn't "can I compose resources" — it's "what happens when I update an RGD that has 100 instances running?"
Imagine you maintain an EKSClusterWithVPC RGD deployed across 100 spoke clusters. You need to introduce a breaking schema change (split image: string into image: {repo: string, tag: string}). How do you migrate the fleet without downtime?

What exists today (v0.9.2)

MechanismWhat it doesWhat it doesn't do
Graph RevisionsImmutable snapshots of each RGD spec change. Keeps last 5. Compilation errors surface on the revision, not on instances.No revision pinning. All instances always use the latest compiled revision.
---------
Breaking change detectionDetects added required fields, removed fields, type changes. Blocks the update by default.No conversion layer. Force-allow = all instances break simultaneously.
kro.run/allow-breaking-changes: "true"Overrides the safety check.No gradual rollout — all instances receive the new schema in the next reconcile tick.

The 100-cluster scenario

1
2
3
`RGD v1: spec.image = "string" ← 100 instances running
RGD v2: spec.image = {repo: "string", tag: "string"} ← breaking change
`
Today's behavior: 1. Apply v2 → kro detects breaking change → blocks (RGD stays on v1 revision) 2. Force with annotation → all 100 instances fail validation next reconcile 3. No way to migrate 10 instances, validate, then migrate the rest

Upstream issues tracking this

IssueTitleStatusWhat it proposes
#482 Allow RGD authors to maintain several versionsOpen, lifecycle/frozenpreviousSchemas[] with conversion webhooks — CRD multi-version pattern
------------
#883 Versioning & rollout controlledOpenMega-feature: canary rollout, revision pinning per instance
#274 Rolling applyOpenApply changes gradually across instances
None are in the v0.10 roadmap. The lifecycle/frozen label on #482 means the maintainers acknowledge the need but haven't committed to a design.

Workarounds for platform teams today

Option A — Version-by-name (safest, most explicit)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
`# Both RGDs coexist
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: ekscluster-v1.kro.run
spec:
schema:
kind: EKSClusterV1
# ...
---
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: ekscluster-v2.kro.run
spec:
schema:
kind: EKSClusterV2
# ...
`
Migrate instance-by-instance: create v2 instance, validate, delete v1 instance. GitOps tracks both.
Option B — Additive-only evolution (no breaking changes ever)
  • Never remove or rename fields
  • New fields always have default= values
  • Use omit() to handle deprecated fields gracefully
  • Eventual deprecation via documentation, not schema enforcement
This is the recommended approach for fleet-scale RGDs until kro gets proper conversion support.
Option C — GitOps progressive rollout (ArgoCD ApplicationSets)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
`apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
generators:
- list:
elements:
- cluster: spoke-1
wave: "1" # migrate first
- cluster: spoke-2
wave: "1"
- cluster: spoke-3
wave: "2" # migrate after validation
template:
spec:
syncPolicy:
syncOptions:
- ApplyOutOfSyncOnly=true
# Wave controls migration order
`
The RGD is identical everywhere, but ArgoCD sync-waves control when each instance gets the updated manifest.
Option D — Conversion webhook (theoretical)
Kubernetes supports conversion webhooks on CRDs. Since kro generates CRDs dynamically from RGDs, there's no stable injection point for a webhook today. This is what #482  proposes to solve natively.

Recommendation for platform teams

For workshop RGDs consumed by multiple teams:
  1. Design schemas as additive-only from day 1 (Option B)
  2. Version-by-name when a breaking change is unavoidable (Option A)
  3. Monitor #482  — if/when kro implements native multi-version RGDs with conversion, it will supersede both workarounds
  4. Use ArgoCD waves (Option C) for controlled fleet rollout timing regardless of schema strategy
⚠️ Bottom line: kro v0.9 has the detection infrastructure for versioning (Graph Revisions, breaking change checks) but not the migration infrastructure. For fleet-scale usage (one RGD → N teams → M instances), plan your schema evolution strategy upfront.

Closing

kro v0.9 moves the project from "declarative resource bundler" to a lightweight fleet controller: one RGD, one instance, N reactive resources driven by cluster state. The forEach + externalRef selector + omit() + cluster-scoped instance combination gives platform teams the same composition power as a hand-written operator — without writing Go.
For teams already using ACK on EKS, this is the release where kro becomes a credible alternative to Crossplane's Compositions or KubeVela's ComponentDefinitions, with a fraction of the operational overhead.
Start here: * The S3BucketSet and GoldTenantPolicy examples above are the lowest-effort way to demonstrate the new model * All manifests + test plan are in the companion gist  * kro-cli validate catches RGD errors before you apply them to a cluster
Maturity reminder: kro's CRD is kro.run/v1alpha1 — the API contract may still introduce breaking changes across minor versions. The project itself is pre-1.0 and actively evolving. Start in workshop/dev clusters (kind, self-managed EKS), and use EKS Capabilities  (managed kro add-on) when it ships the version you need for production. Note that EKS Capabilities does not let you choose the kro version — AWS controls the release cadence.

References

  • Release notes v0.9.0: https://github.com/kubernetes-sigs/kro/releases/tag/v0.9.0
  • Release notes v0.9.2: https://github.com/kubernetes-sigs/kro/releases/tag/v0.9.2
  • Collections (forEach): https://kro.run/docs/concepts/rgd/resource-definitions/collections
  • External references: https://kro.run/docs/concepts/rgd/resource-definitions/external-references
  • Cluster-scoped schema: https://kro.run/docs/concepts/rgd/schema#scope
  • Conditional inclusion: https://kro.run/docs/concepts/rgd/resource-definitions/conditional-inclusion
  • CEL libraries: https://kro.run/docs/concepts/cel/cel-libraries
  • Feature gates: https://kro.run/docs/advanced/feature-gates
  • Graph Revisions: https://kro.run/docs/advanced/graph-revisions
  • RBAC / Access Control: https://kro.run/docs/advanced/access-control/
  • EKS Capabilities (managed kro, version not selectable): https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
  • Platform Engineering on EKS workshop: https://catalog.workshops.aws/platform-engineering-on-eks/en-US
  • Sample RGDs (appmod-blueprints): https://github.com/aws-samples/appmod-blueprints
  • SimpleSchema spec: https://kro.run/api/specifications/simple-schema
  • kro roadmap (KREPs): https://kro.run/roadmap
  • KubeVela (OAM): https://kubevela.io/docs/
  • kro issue #1290 (field ownership): https://github.com/kubernetes-sigs/kro/issues/1290
  • kro issue #1048 (extension/traits): https://github.com/kubernetes-sigs/kro/issues/1048
  • kro issue #1293 (RGD composition): https://github.com/kubernetes-sigs/kro/issues/1293
  • kro issue #482 (multi-version RGD): https://github.com/kubernetes-sigs/kro/issues/482
  • kro issue #883 (versioning & rollout): https://github.com/kubernetes-sigs/kro/issues/883
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article