
kro v0.9 Deep Dive: omit(), Reconciliation Limits, and Enterprise Readiness
Part 2 of our kro v0.9 exploration: the omit() function for field-level conditionals without duplication, reconciliation behavior with HPA/KEDA/Argo Rollouts (validated compatibility matrix), enterprise gap analysis vs KubeVela, and Day 2 RGD versioning strategies for fleet-scale deployments.
> This is Part 2 of our kro v0.9 exploration. Part 1 covers forEach collections, externalRef selectors, cluster-scoped instances, CEL enrichments, and Graph Revisions.
The omit() function: solving composition without duplication
kro v0.9.0 introduced a feature-gated CEL function called
omit() that solves one of the most painful composition problems: field-level conditional inclusion.The problem
In complex applications like in the appmod-blueprints reference architecture, where the
appmod-service.yaml RGD needs to support multiple optional capabilities: ingress, blue/green rollouts, HPA autoscaling, IAM role annotations, etc. Without field-level conditionals, you face a combinatorial explosion: with N optional features, you need up to 2^N near-identical resource blocks using includeWhen at the resource level. Any structural change must be replicated across every variant.The solution: omit()
Enable the feature gate:
1
2
3
4
5
`# Helm values
config:
featureGates:
CELOmitFunction: true
`Or pass the flag directly:
1
2
`--feature-gates=CELOmitFunction=true
`Then use
omit() in CEL expressions to conditionally remove a field entirely:1
2
3
4
`apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
# ... (full manifest in companion gist)Why this matters: Server-Side Apply field ownership
When
omit() drops a field, kro does not take ownership of that field via Server-Side Apply. We validated this on a kind cluster:1
2
3
4
5
6
7
8
`# With omit() → kro does NOT own replicas
Manager: kro.run/applyset
→ replicas NOT managed ✅
# Without omit() → kro DOES own replicas
Manager: kro.run/applyset
→ replicas IS managed ❌ (reset on every reconcile)
`This means: - A field removed by
omit() can be freely managed by HPA, KEDA, Argo Rollouts, or any external controller - kro will never reset it back, because it never claimed ownership - The field is still created with K8s API defaults (e.g., replicas defaults to 1)Combining omit() with includeWhen
The combination of both mechanisms gives you full composition control in a single RGD — no duplication:
| Mechanism | Scope | Use case |
|---|---|---|
includeWhen | Entire resource | Optional Ingress, HPA, NetworkPolicy |
| --- | --- | --- |
omit() | Single field | Optional annotation, replicas delegation, conditional selector |
A real-world example — one RGD, four deployment profiles:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
`resources:
- id: deployment
template:
# ...
spec:
replicas: "${schema.spec.replicas \> 0 ? schema.spec.replicas : }"
# ...
- id: ingress
includeWhen:
- "${schema.spec.enableIngress}"
template:
# ... only created when ingress is enabled
- id: hpa
includeWhen:
- "${schema.spec.enableHPA}"
template:
# ... only created when HPA is enabled
`Results from our kind cluster tests:
| Instance | replicas set? | Ingress | HPA | Actual replicas |
|---|---|---|---|---|
| minimal (replicas=0, no ingress, no HPA) | omitted | ❌ | ❌ | 1 (K8s default) |
| --- | --- | --- | --- | --- |
| full (replicas=0, ingress=true, HPA=true) | omitted | ✅ | ✅ min=2 | 2 (HPA min) |
| hpa-only (replicas=0, HPA=true) | omitted | ❌ | ✅ min=3 | 3 (HPA min) |
| fixed (replicas=5) | 5 | ❌ | ❌ | 5 (kro-managed) |
Solving the reconciliation problem with omit()
Remember the reconciliation limits? The previous workaround was to simply not put the field in the RGD template at all. But what if you want the field to be configurable — sometimes set by the user, sometimes delegated to an external controller?
omit() bridges this gap:1
2
3
4
`# User sets replicas=3 → kro manages it (field IS in the SSA payload)
# User sets replicas=0 → kro omits it (field NOT in the SSA payload → HPA free to scale)
replicas: "${schema.spec.replicas \> 0 ? schema.spec.replicas : }"
`This is the missing piece between "always managed by kro" and "never in the template". It gives the instance author the choice, per-deployment.
Limitations and caveats
- Alpha feature gate —
CELOmitFunctionmust be explicitly enabled. Not available on EKS Capabilities (managed kro) yet. - Scalar fields only —
omit()removes the containing field. You cannot conditionally omit a nested block (e.g., an entirestrategy:section). For that, useincludeWhenon a separate resource. - Required metadata fields —
omit()is rejected onmetadata.name,metadata.namespace,apiVersion,kind. - CEL type safety — The return type of the ternary must match the field type.
omit()is polymorphic but the "truthy" branch must be the correct type.
Reproduce it yourself: Enable the feature gate, apply the manifests from the companion gist (
rg-webapp-omit.yaml), and verify with kubectl get deploy -o jsonpath='{.metadata.managedFields}'.What's next for kro
The v0.10 milestone has 8 KREPs scheduled. The ones most relevant to platform teams:
- KREP-006 — Propagation Control (
propagateWhen): Gate mutations as changes propagate through the graph. Complementary toreadyWhen— together they bookend when a resource can start mutating and when it's considered complete. This will give finer control over rollout ordering in multi-resource RGDs. - KREP-011 — RGD Variables (
dataresources): Define computed data inside an RGD that isn't a Kubernetes resource — think shared constants, computed ARNs, or intermediate values reusable across templates without creating a ConfigMap. - KREP-021 — kro CLI: A dedicated CLI for validating, testing, and managing RGDs locally before applying them to a cluster.
- KREP-014 — Resource Lifecycles: Control what happens when resources are removed from an RGD — delete, orphan, or retain.
- KREP-005 — Level-based Topological Sorting: Smarter parallelism in resource creation based on graph depth levels.
- Detach on deletion (#183 ,
lifecycle/frozen): Delete an RGD instance while keeping its child resources alive — proposed annotationkro.run/detach-on-deletion: "true". Not yet implemented. Note:kro.run/reconcile: disabled(#1191 ) pauses reconciliation but does NOT prevent cascade deletion — the finalizer still fires onkubectl delete.
Further out, v1.0 is on the roadmap but has no KREPs assigned yet — expect API stabilization and graduation from
v1alpha1 to happen there.Follow progress on the kro roadmap page or the #kro Slack channel .
Reconciliation limits: what kro resets and what it leaves alone
kro reconciles resources using a strategic merge patch — which means it only touches fields that appear in the RGD template. This has profound implications when kro-managed resources interact with other controllers (HPA, KEDA, Argo Rollouts, sidecar injectors).
We validated these behaviors on a kind cluster with kro v0.9.2 and filed them upstream as kubernetes-sigs/kro#1290 .
Reproduce it yourself: The companion gist contains all manifests (
rg-webapp.yaml, rg-webapp-hpa-compatible.yaml, webhook injector) and a step-by-step test plan (TESTING-RECONCILIATION.md) you can run on a local kind cluster in ~5 minutes.What kro resets (❌ conflicts)
| Controller | Field affected | Behavior |
|---|---|---|
| HPA | deployment.spec.replicas | kro resets to RGD value in <2s |
| --- | --- | --- |
| KEDA ScaledObject | deployment.spec.replicas (via KEDA-managed HPA) | kro resets in <3s |
| Argo Rollouts | service.spec.selector (adds hash label) | kro removes the hash immediately |
| Any controller modifying a field present in the RGD template | That field | Reset on every reconcile loop (~1-3s) |
The root cause: kro treats every field in the RGD template as authoritative. If
replicas: 2 is in your RGD, kro will always set it back to 2, regardless of what the HPA decides.What kro leaves alone (✅ coexistence)
| Mutation | Where it happens | Behavior |
|---|---|---|
| Pod-level sidecar injection (Istio, Linkerd, custom webhooks) | MutatingWebhook on Pod CREATE | ✅ kro manages the Deployment template; the webhook mutates Pods at creation — no conflict |
| --- | --- | --- |
| initContainers added to Deployment template | Webhook or external controller patches the Deployment | ✅ Persists indefinitely — kro doesn't specify initContainers in the template, so it ignores them |
| Extra containers added to Deployment template | Same pattern (Vault Agent, envoy sidecar) | ✅ Persists — fields absent from the RGD template are not reconciled |
| Annotations/labels added by external systems | Any controller or webhook | ✅ Persists — as long as they're not in the RGD template |
The key insight: kro's merge behavior is additive for fields it doesn't own, and authoritative for fields it does. If you don't put it in the template, kro won't touch it.
Workarounds for existing RGDs
Problem: You want HPA/KEDA autoscaling on a Deployment managed by kro.
Solution: Don't put
replicas in the RGD template. Let it default via the Kubernetes API server (which defaults to 1), then let HPA/KEDA manage it:1
2
3
4
`# ❌ BAD — kro will fight the HPA
resources:
- id: deployment
# ... (full manifest in companion gist)Problem: You use Argo Rollouts which adds a
-hash label to Service selectors during canary/blue-green.Solution (validated ✅): Don't specify the
selector in the Service template. Argo Rollouts will inject its own rollouts-pod-template-hash selector at runtime, and kro will leave it alone since it's not in the RGD. We tested a full blue/green cycle (deploy → preview → promote → abort/rollback) with kro v0.9.2 and Argo Rollouts v1.7.2 — zero conflicts.Problem: You want Vault Agent or Istio to inject sidecars.
Solution: No workaround needed! Pod-level webhooks coexist naturally. Even Deployment-level mutations (initContainers, extra containers) persist as long as they add fields not present in the RGD template.
Compatibility matrix
| Pattern | Compatible with kro? | Condition |
|---|---|---|
| HPA / KEDA / VPA | ✅ Yes | Remove replicas from RGD template |
| --- | --- | --- |
| Argo Rollouts (blue/green) | ✅ Yes | Don't specify Service selector in RGD — Rollouts injects its hash-based selector at runtime |
| Istio / Linkerd sidecar | ✅ Yes | No changes needed |
| Vault Agent Injector | ✅ Yes | No changes needed |
| Kyverno mutating policies | ✅ Yes | As long as they add/modify fields not in the RGD template |
| Custom admission webhooks | ✅ Yes | Same rule — don't mutate fields kro owns |
Future fix: The kro team is tracking
ignoreFields / managementPolicy proposals that would let you explicitly delegate ownership of specific fields to other controllers. Until then, the workaround is structural: keep runtime-mutable fields out of your RGD templates.Composition & Ordering
kro v0.9's graph-based reconciliation unlocks two composition features that matter for platform teams: ordered resource creation and collection iteration.
readyWhen — Execution ordering via dependency gates
kro has no explicit
dependsOn field. Instead, ordering is inferred from CEL references between resources. If resource B references ${a.metadata.name}, kro places B after A in the topological sort. Combined with readyWhen, this creates proper deployment sequencing:1
2
3
4
`resources:
- id: config
template:
# ... (full manifest in companion gist)Observed behavior (validated on kind): - ConfigMap + Deployment created at T+0 (ConfigMap's
readyWhen is instantly satisfied) - Service created at T+7s — only after Deployment reached readyReplicas \>= 1 - Topological order: ["config", "deployment", "service"]Key insight: The
readyWhen gate only delays downstream creation when the expression takes time to become true. If your upstream readyWhen is instantly satisfied (e.g., checking name != ''), kro creates both in the same reconcile cycle.Syntax pitfalls (discovered via testing): -
readyWhen can only reference the resource's own status fields — not schema.* - Status fields in the RGD schema must be CEL expressions referencing resources, not bare type declarations - Use string() for type casting when injecting numeric/int values into string annotationsforEach — Collection iteration with custom types
forEach expands a single resource definition into N instances from a list. The syntax requires custom types for object arrays:1
2
3
4
`spec:
schema:
apiVersion: v1alpha1
# ... (full manifest in companion gist)Edge cases validated (all passed ✅ on kind v0.9.2):
| Case | Result |
|---|---|
Empty array ([]) | Instance ACTIVE immediately, 0 resources |
| --- | --- |
| Single item | 1 namespace + 1 deployment |
| 20 items | 20 NS + 20 deploys in ~30s |
| Add item dynamically | Resources created |
| Remove items (4→2) | Orphaned resources deleted cleanly |
| Modify replicas in-place | Updated without recreating |
Syntax pitfalls: -
forEach must be an array of {varName: "${CEL}"} objects (not a bare CEL string) - Array of objects in the schema requires a types: section — inline []{} notation is not supported - Cartesian product (multiple forEach dimensions) is supported but untestedreadyWhen + forEach + omit() + includeWhen = Full composition toolkit
With v0.9.2, kro now covers four orthogonal composition axes:
| Axis | Mechanism | Scope |
|---|---|---|
| Resource-level conditional | includeWhen | Include/exclude entire resources |
| --- | --- | --- |
| Field-level conditional | omit() (CELOmitFunction) | Include/exclude individual fields |
| Iteration | forEach + custom types | N instances from a list |
| Ordering | CEL ref graph + readyWhen | Sequential creation with health gates |
This means a single RGD can express: "For each team in the list, create a namespace, optionally create an HPA only if
enableHPA is true, omit the replicas field when HPA is active, and don't create the Ingress until the Service is ready."Enterprise gaps — kro vs KubeVela requirements
After extensive testing of kro v0.9.2 against OAM/KubeVela patterns , here's a gap analysis for enterprise platform teams. Each requirement is mapped to existing kro solutions or upstream proposals.
| # | Requirement | KubeVela equivalent | kro v0.9 status | Upstream issue/KREP |
|---|---|---|---|---|
| 1 | Modular composition — import RGD modules, compose RGDs from smaller RGDs | Components + Application | ❌ Not supported — nested RGDs exist but have reconciliation loop bugs | #1293 (deferred schema), #1057 |
| --- | --- | --- | --- | --- |
| 2 | Declarative permission/policy — conditional IAM annotations, policy injection | Traits (labels/annotations trait) | ✅ Solved — omit() for conditional annotations (iamRoleArn != '' ? val : omit()) | #1290 |
| 3 | Apply-once / field management — don't fight external controllers | Trait takeover mode | ✅ Solved — omit() removes SSA ownership; external controllers manage freely | #1290 |
| 4 | Custom health checks & ordering — wait for cert-manager, operator CRD readiness | Health probes in workflow | ⚠️ Partial — readyWhen works but limited to resource's own fields. No HTTP probes, no shell exec | #542 (lifecycle modifiers) |
| 5 | Global application context — cluster name, region, account ID accessible to all RGDs | Application context | ⚠️ Partial — externalRef can read ConfigMaps/Secrets. No built-in global context injection | #920 (Global Variables, needs-krep) |
| 6 | Iteration/looping — for each tenant, for each region | ComponentDefinition with for | ✅ Solved — forEach with custom types, dynamic add/remove, 0-N items, cartesian product | Already in v0.9 |
| 7 | Traits system — cross-cutting concerns (sidecar injection, observability, network policies) | Traits (TraitDefinition) | ❌ Not supported — no mechanism to apply cross-cutting mutations across multiple RGDs | #1048 (Extension mechanism, stale) |
| 8 | Controller stability — no reconciliation fights, predictable behavior under load | N/A (stable since v1.0) | ⚠️ Partial — SSA-based, predictable for managed fields. But any field in template gets force-reconciled (known) | — |
| 9 | Field-level conditionals — without duplicating entire resources | Patch trait with conditions | ✅ Solved — omit() on scalars + includeWhen on resources. Alpha, scalar-only limitation | #928 |
Summary: 4 solved, 3 partial, 2 gaps
Solved (4/9): Field management (#2, #3), iteration (#6), field-level conditionals (#9)
Partial (3/9): Health checks (#4 —
readyWhen covers 80% of cases), global context (#5 — externalRef workaround), stability (#8 — predictable but opinionated)Gaps (2/9): Modular composition (#1), traits (#7)
The two remaining gaps (#1 modular composition, #7 traits) are architectural — they require new abstractions at the kro controller level. For platform teams today: - #1 workaround: One "fat" RGD per team type (vs. composing small reusable blocks) - #7 workaround: Kyverno/Gatekeeper policies + admission webhooks for cross-cutting concerns
Upcoming KREPs addressing gaps (v0.10 roadmap)
| KREP | Title | Addresses |
|---|---|---|
| KREP-006 | Propagation Control (propagateWhen) | #4 partially — gates when mutations propagate |
| --- | --- | --- |
| KREP-011 | RGD Variables (data resources) | #5 — complex data without K8s resources |
| KREP-014 | Branch-aware dependency inference | #4 — smarter ordering with conditional branches |
| #920 | Global Variables (needs-krep) | #5 — cluster-level context injection |
| #892 | RBAC isolation for RGDs (needs-krep) | #2 — permission boundaries |
Day 2: RGD versioning & progressive migration
The biggest production question isn't "can I compose resources" — it's "what happens when I update an RGD that has 100 instances running?"
Imagine you maintain an
EKSClusterWithVPC RGD deployed across 100 spoke clusters. You need to introduce a breaking schema change (split image: string into image: {repo: string, tag: string}). How do you migrate the fleet without downtime?What exists today (v0.9.2)
| Mechanism | What it does | What it doesn't do |
|---|---|---|
| Graph Revisions | Immutable snapshots of each RGD spec change. Keeps last 5. Compilation errors surface on the revision, not on instances. | No revision pinning. All instances always use the latest compiled revision. |
| --- | --- | --- |
| Breaking change detection | Detects added required fields, removed fields, type changes. Blocks the update by default. | No conversion layer. Force-allow = all instances break simultaneously. |
kro.run/allow-breaking-changes: "true" | Overrides the safety check. | No gradual rollout — all instances receive the new schema in the next reconcile tick. |
The 100-cluster scenario
1
2
3
`RGD v1: spec.image = "string" ← 100 instances running
RGD v2: spec.image = {repo: "string", tag: "string"} ← breaking change
`Today's behavior: 1. Apply v2 → kro detects breaking change → blocks (RGD stays on v1 revision) 2. Force with annotation → all 100 instances fail validation next reconcile 3. No way to migrate 10 instances, validate, then migrate the rest
Upstream issues tracking this
| Issue | Title | Status | What it proposes |
|---|---|---|---|
| #482 | Allow RGD authors to maintain several versions | Open, lifecycle/frozen | previousSchemas[] with conversion webhooks — CRD multi-version pattern |
| --- | --- | --- | --- |
| #883 | Versioning & rollout controlled | Open | Mega-feature: canary rollout, revision pinning per instance |
| #274 | Rolling apply | Open | Apply changes gradually across instances |
None are in the v0.10 roadmap. The
lifecycle/frozen label on #482 means the maintainers acknowledge the need but haven't committed to a design.Workarounds for platform teams today
Option A — Version-by-name (safest, most explicit)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
`# Both RGDs coexist
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: ekscluster-v1.kro.run
spec:
schema:
kind: EKSClusterV1
# ...
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
name: ekscluster-v2.kro.run
spec:
schema:
kind: EKSClusterV2
# ...
`Migrate instance-by-instance: create v2 instance, validate, delete v1 instance. GitOps tracks both.
Option B — Additive-only evolution (no breaking changes ever)
- Never remove or rename fields
- New fields always have
default=values - Use
omit()to handle deprecated fields gracefully - Eventual deprecation via documentation, not schema enforcement
This is the recommended approach for fleet-scale RGDs until kro gets proper conversion support.
Option C — GitOps progressive rollout (ArgoCD ApplicationSets)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
`apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
spec:
generators:
- list:
elements:
- cluster: spoke-1
wave: "1" # migrate first
- cluster: spoke-2
wave: "1"
- cluster: spoke-3
wave: "2" # migrate after validation
template:
spec:
syncPolicy:
syncOptions:
- ApplyOutOfSyncOnly=true
# Wave controls migration order
`The RGD is identical everywhere, but ArgoCD sync-waves control when each instance gets the updated manifest.
Option D — Conversion webhook (theoretical)
Kubernetes supports conversion webhooks on CRDs. Since kro generates CRDs dynamically from RGDs, there's no stable injection point for a webhook today. This is what #482 proposes to solve natively.
Recommendation for platform teams
For workshop RGDs consumed by multiple teams:
- Design schemas as additive-only from day 1 (Option B)
- Version-by-name when a breaking change is unavoidable (Option A)
- Monitor #482 — if/when kro implements native multi-version RGDs with conversion, it will supersede both workarounds
- Use ArgoCD waves (Option C) for controlled fleet rollout timing regardless of schema strategy
⚠️ Bottom line: kro v0.9 has the detection infrastructure for versioning (Graph Revisions, breaking change checks) but not the migration infrastructure. For fleet-scale usage (one RGD → N teams → M instances), plan your schema evolution strategy upfront.
Closing
kro v0.9 moves the project from "declarative resource bundler" to a lightweight fleet controller: one RGD, one instance, N reactive resources driven by cluster state. The
For teams already using ACK on EKS, this is the release where kro becomes a credible alternative to Crossplane's Compositions or KubeVela's ComponentDefinitions, with a fraction of the operational overhead.
Start here: * The
Maturity reminder: kro's CRD is
forEach + externalRef selector + omit() + cluster-scoped instance combination gives platform teams the same composition power as a hand-written operator — without writing Go.For teams already using ACK on EKS, this is the release where kro becomes a credible alternative to Crossplane's Compositions or KubeVela's ComponentDefinitions, with a fraction of the operational overhead.
Start here: * The
S3BucketSet and GoldTenantPolicy examples above are the lowest-effort way to demonstrate the new model * All manifests + test plan are in the companion gist * kro-cli validate catches RGD errors before you apply them to a clusterMaturity reminder: kro's CRD is
kro.run/v1alpha1 — the API contract may still introduce breaking changes across minor versions. The project itself is pre-1.0 and actively evolving. Start in workshop/dev clusters (kind, self-managed EKS), and use EKS Capabilities (managed kro add-on) when it ships the version you need for production. Note that EKS Capabilities does not let you choose the kro version — AWS controls the release cadence.References
- Release notes v0.9.0: https://github.com/kubernetes-sigs/kro/releases/tag/v0.9.0
- Release notes v0.9.2: https://github.com/kubernetes-sigs/kro/releases/tag/v0.9.2
- Collections (
forEach): https://kro.run/docs/concepts/rgd/resource-definitions/collections - External references: https://kro.run/docs/concepts/rgd/resource-definitions/external-references
- Cluster-scoped schema: https://kro.run/docs/concepts/rgd/schema#scope
- Conditional inclusion: https://kro.run/docs/concepts/rgd/resource-definitions/conditional-inclusion
- CEL libraries: https://kro.run/docs/concepts/cel/cel-libraries
- Feature gates: https://kro.run/docs/advanced/feature-gates
- Graph Revisions: https://kro.run/docs/advanced/graph-revisions
- RBAC / Access Control: https://kro.run/docs/advanced/access-control/
- EKS Capabilities (managed kro, version not selectable): https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
- Platform Engineering on EKS workshop: https://catalog.workshops.aws/platform-engineering-on-eks/en-US
- Sample RGDs (appmod-blueprints): https://github.com/aws-samples/appmod-blueprints
- SimpleSchema spec: https://kro.run/api/specifications/simple-schema
- kro roadmap (KREPs): https://kro.run/roadmap
- KubeVela (OAM): https://kubevela.io/docs/
- kro issue #1290 (field ownership): https://github.com/kubernetes-sigs/kro/issues/1290
- kro issue #1048 (extension/traits): https://github.com/kubernetes-sigs/kro/issues/1048
- kro issue #1293 (RGD composition): https://github.com/kubernetes-sigs/kro/issues/1293
- kro issue #482 (multi-version RGD): https://github.com/kubernetes-sigs/kro/issues/482
- kro issue #883 (versioning & rollout): https://github.com/kubernetes-sigs/kro/issues/883
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article