AWS Builder Center
OKF: giving your AI agents a filing cabinet instead of a search engine

OKF: giving your AI agents a filing cabinet instead of a search engine

Most AI agents fail on internal knowledge, not reasoning. Google's Open Knowledge Format (OKF) fixes this by packaging schemas, metrics, and runbooks as plain markdown files with YAML frontmatter, no database, no SDK, no vendor lock-in. This post covers where OKF came from, what a concept file actually looks like, when it's worth adopting, working Python code for reading and validating a bundle, an architecture diagram, and where RAG is still the better choice over curated markdown.

OKF: giving your AI agents a filing cabinet instead of a search engine

If you have spent any time building agents against your own company's data, you have probably hit the same wall over and over. The model is smart enough to write the query, but it has no idea that "active users" excludes refunded orders, or that the customers table joins on  customer_id  and not  email , or that the runbook for a failed nightly job lives in a Confluence page nobody has opened since March. That knowledge exists somewhere in your organization. It's just scattered across wikis, Slack threads, code comments, and the heads of two senior engineers who are both on vacation this week.
Google Cloud's Data Cloud team ran into this same problem while building agents on top of BigQuery, and in June 2026 they published something to fix it: the Open Knowledge Format, or OKF. It's not a product and not a service. It's a spec for writing down what your organization knows in a way that both people and agents can read without a translation layer in between.
This post walks through where OKF came from, what it actually is, when it earns its place in your stack, how to use it with working code, and when you are better off reaching for retrieval augmented generation instead.

Where OKF came from

To understand why OKF looks the way it does, it helps to know the shape of the problem it was responding to.
Foundation models had gotten good enough to write code, summarize documents, and reason over structured data, but they kept failing in the same spot: they didn't know the internal facts about a company's systems. Not because the facts were secret, but because the facts lived in a dozen incompatible places. A metadata catalog has its own API. A wiki has its own export format. A senior engineer's mental model of how two systems join together doesn't have an export format at all.
Sam McVeety and Amir Hormati, tech leads on the Data Cloud team, noticed that a pattern kept showing up organically to solve this. People were already writing markdown files with a bit of YAML on top and letting an agent read them directly, no database, no API, just files. Andrej Karpathy had described something close to this in a gist about LLM wikis, pointing out that language models don't get bored updating cross references the way humans do, so a wiki that a person would eventually abandon out of sheer tedium is exactly the kind of thing a model is happy to maintain.
The trouble was that every team's version of this pattern was a little different. One team's frontmatter had a  type  field, another's didn't. One team linked concepts with relative paths, another used slugs. None of it was portable. If you built an agent against your team's wiki, it broke the moment you pointed it at a different team's wiki, even though both were "just markdown with frontmatter."
OKF is Google's attempt to take that organic pattern and pin down the small number of conventions needed to make it interoperable. Not a new invention, more of a standardization of something a lot of people were already doing slightly differently.

What OKF actually is

At its core, OKF represents a piece of organizational knowledge, whether that's a database table, a metric definition, an API, or a runbook, as one markdown file called a concept. A directory full of these files is called a bundle, and the file path itself is the concept's identity. There's no separate ID scheme to keep in sync with the filesystem.
Each concept file has two parts: a small YAML frontmatter block for the handful of fields that need to be searchable or filterable, and a markdown body for everything else, tables, code blocks, prose, whatever the concept needs.
sales/
├── index.md
├── datasets/
│ └── orders_db.md
├── tables/
│ ├── orders.md
│ └── customers.md
└── metrics/
└── weekly_active_users.md
And here's what one of those files looks like in practice:
---
type: BigQuery Table
title: Orders
description: One row per completed customer order.
resource: https://console.cloud.google.com/bigquery?p=acme&d=sales&t=orders
tags: [sales, revenue]
timestamp: 2026-05-28T14:30:00Z
---
# Schema

| Column | Type | Description |
|----------------|--------|-------------------------------------------|
| `order_id` | STRING | Globally unique order identifier. |
| `customer_id` | STRING | FK to [customers](../tables/customers.md).|

# Joins

Joined with [customers](../tables/customers.md) on `customer_id`.
The only field the spec actually requires is  type . Everything else, what types your organization uses, what other fields you add, how you structure the body, is left up to you. This is a deliberate choice. OKF is trying to standardize the interoperability surface, not the content model, so it doesn't tell you how to organize your knowledge, only how to package it so someone else's tooling can read it without asking you what your schema means first.
The markdown links between files matter more than they look. Because  orders.md  links to  customers.md , and the metric file links back to  orders.md , the bundle isn't really a tree, it's a graph. An agent walking the bundle can follow those links the same way a person would click through a wiki, discovering that the metric depends on the table, which depends on the join, without anyone having built a separate graph database to hold that relationship.
Here's a quick comparison of how this differs from the metadata catalog approach most data teams already have:

Traditional metadata catalog
OKF bundle
Storage
Proprietary database behind an API
Plain files in a directory
Access
SDK or REST calls
Any file reader, any git client
Versioning
Whatever the vendor bolts on
Native git history
Vendor lock-in
High, tied to the catalog product
None, it's just markdown
Editable by hand
Rarely, goes through the UI
Yes, any text editor
Cross-org portability
Usually requires export/import tooling
Copy the directory
Agent readability
Needs an integration per catalog
Needs a markdown parser

When to use it

OKF earns its keep when the knowledge you're trying to capture is relatively stable, high value, and currently locked inside something an agent can't reach on its own. Think schema documentation, metric definitions, runbooks, deprecation notices, and the tribal knowledge that normally lives in one person's head. If losing access to that one person for a week would hurt your team's ability to answer basic questions about your own systems, that's a strong candidate for an OKF bundle.
It's also a good fit when you want the knowledge itself to be portable. If you're building a tool that other teams, or other companies, might want to plug their own agents into, shipping your context as an OKF bundle means anyone can consume it without adopting your specific catalog product or writing a custom integration against your API.
It's less obviously useful for information that changes every few minutes, or information so voluminous that no human would ever want to skim it directly. That's the boundary we'll get into in the RAG section.

How to use it

Getting a bundle from zero to something an agent can rely on has three practical steps: write the concepts, validate the bundle, then read it back out programmatically. None of this needs a server or a database, which is most of the point.

Writing a concept by hand

A concept file is nothing more than frontmatter plus markdown, so writing one is close to writing documentation you'd want a teammate to read anyway.
---
type: Metric
title: Weekly Active Users
description: Distinct customers with at least one completed order in a trailing 7 day window.
tags: [sales, engagement]
timestamp: 2026-06-01T10:00:00Z
---
# Definition

​```sql
SELECT
COUNT(DISTINCT customer_id) AS weekly_active_users
FROM `acme.sales.orders`
WHERE order_ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
​```

# Source table

Computed from [orders](../tables/orders.md).

# Notes

Refunded orders are not excluded. If you need net WAU, join against the
refunds table before counting distinct customers.
Notice the "Notes" section. This is where OKF earns the "curated insight" part of its description. A metadata catalog will tell you the column types. It won't tell you that refunds aren't excluded from this particular metric, because that's a fact that lives in a person's head until someone bothers to write it down. That gap is exactly what OKF is for.

Reading a bundle back out

An agent, or any other consumer, needs to walk the directory, pull the frontmatter out of each file, and follow the links to build up the relationships between concepts. Here's a small reader that does that:
import re
from dataclasses import dataclass, field
from pathlib import Path

import yaml

LINK_PATTERN = re.compile(r"\[([^\]]+)\]\(([^)]+\.md)\)")
FRONTMATTER_PATTERN = re.compile(r"^---\n(.*?)\n---\n(.*)$", re.DOTALL)

@dataclass
class Concept:
path: Path
type: str
title: str
description: str = ""
tags: list = field(default_factory=list)
body: str = ""
links_to: list = field(default_factory=list)

def parse_concept(path: Path) -> Concept:
raw = path.read_text(encoding="utf-8")
match = FRONTMATTER_PATTERN.match(raw)
if not match:
raise ValueError(f"{path} has no YAML frontmatter, not a valid OKF concept")

frontmatter, body = match.groups()
meta = yaml.safe_load(frontmatter) or {}

if "type" not in meta:
raise ValueError(f"{path} is missing the required 'type' field")

links = [href for _, href in LINK_PATTERN.findall(body)]

return Concept(
path=path,
type=meta["type"],
title=meta.get("title", path.stem),
description=meta.get("description", ""),
tags=meta.get("tags", []),
body=body.strip(),
links_to=links,
)

def load_bundle(root: str) -> dict[str, Concept]:
root_path = Path(root)
concepts = {}
for md_file in root_path.rglob("*.md"):
rel = md_file.relative_to(root_path).as_posix()
concepts[rel] = parse_concept(md_file)
return concepts
Running this against the sample bundle from earlier prints out each concept along with what it links to:
$ python3 okf_reader.py
metrics/weekly_active_users.md: Metric : Weekly Active Users
links to: ['../tables/orders.md']
tables/customers.md: BigQuery Table : Customers
links to: ['../tables/orders.md']
tables/orders.md: BigQuery Table : Orders
links to: ['../tables/customers.md', '../tables/customers.md', '../metrics/weekly_active_users.md']
That's the whole graph, built with a regex and a YAML parser. No vector database, no embedding step, no service to stand up.

Validating a bundle before you commit it

Because a bundle is just files, it's easy for links to drift or rot the same way any documentation does. A small conformance check catches the obvious problems before they land in git, the same way a linter catches syntax errors before a build:
from collections import defaultdict
from pathlib import Path

from okf_reader import load_bundle

def validate(root: str) -> list[str]:
problems = []
concepts = load_bundle(root)
known_paths = set(concepts.keys())
seen_titles = defaultdict(list)
root_abs = Path(root).resolve()

for key, concept in concepts.items():
if not concept.type:
problems.append(f"{key}: missing required 'type' field")

seen_titles[(concept.type, concept.title)].append(key)

for link in concept.links_to:
concept_dir_abs = root_abs / Path(key).parent
resolved = (concept_dir_abs / link).resolve().relative_to(root_abs).as_posix()
if resolved not in known_paths:
problems.append(f"{key}: broken link to '{link}' (resolved to '{resolved}')")

for (type_, title), keys in seen_titles.items():
if len(keys) > 1:
problems.append(f"duplicate '{type_}' concept titled '{title}': {keys}")

return problems
Point it at a good bundle and it comes back clean:
$ python3 okf_validate.py sample_bundle
sample_bundle: valid, no conformance issues found
Break a link on purpose and it catches it immediately:
$ python3 okf_validate.py sample_bundle_broken
sample_bundle_broken: 2 issue(s) found
- tables/orders.md: broken link to '../tables/nope.md' (resolved to 'tables/nope.md')
- tables/orders.md: broken link to '../tables/nope.md' (resolved to 'tables/nope.md')
Wire this into a pre-commit hook or a CI step and your bundle stays trustworthy the same way tests keep your codebase trustworthy.

How the pieces fit together

Here's the full picture, from the people and pipelines that write concepts, through the bundle itself, to the agents and tools that consume it.
The bottom strip in that diagram is worth pausing on, because it's the whole subject of the next section.

When not to use it, and where RAG fits better

OKF is a poor fit the moment your knowledge stops being something a human would want to curate. If you're trying to give an agent access to millions of support tickets, a decade of Slack history, or a corpus of legal documents too large for anyone to realistically read and organize into concept files, writing OKF by hand isn't going to happen, and generating it automatically just turns your bundle into a low-quality index with extra YAML on top.
This is where retrieval augmented generation is the better tool. RAG doesn't ask anyone to curate anything up front. You embed your documents into a vector store, and at query time you pull back whatever chunks are most similar to the question being asked. It's built for volume and for content that's too large, too unstructured, or too fast-moving for a person to maintain by hand.
The two approaches also fail differently, which matters more than it sounds like it should. An OKF bundle is only as good as its last edit, so it can go stale if nobody updates it, but when you read a concept, you know exactly what you're getting, a fact someone deliberately wrote down and can be held accountable for. RAG is always fresh relative to whatever's in the corpus, but similarity search means you can retrieve a chunk that's topically related and still wrong for the specific question, and there's no author standing behind that chunk the way there is with a curated concept file.

OKF
RAG
Best for
Stable, high value, curated facts
Large, high volume, fast changing corpora
Who maintains it
A person or a producer pipeline, deliberately
Nobody, it indexes whatever exists
Freshness
As fresh as the last edit
As fresh as the last re-index
Failure mode
Goes stale if unmaintained
Retrieves plausible but wrong chunks
Infra required
None, just files
Vector store, embedding pipeline
Trust model
You know who wrote each fact
Similarity score, not authorship
Good example
"How is weekly active users defined"
"Find every ticket that mentions this error"
In practice these two are not really competing for the same job, and the most capable agent setups use both. An agent might read an OKF bundle to get the two or three facts that are load bearing, exactly how a metric is defined, which table is the source of truth, then fall back to RAG over a much larger corpus when the question needs something that curated markdown was never going to hold. Treating them as an either-or choice usually means picking the wrong tool for whichever half of the problem you deprioritized.

Wrapping up

OKF isn't trying to replace your metadata catalog or your vector database. It's filling a gap that neither of those tools was built for: giving agents access to the small set of facts about your systems that are worth a human deliberately writing down and standing behind. The bar for reaching for it is whether the knowledge is stable enough and important enough that someone should own it as a file in git, not whether you have a lot of data lying around.
If you're building agents against internal systems today, a good first move is picking three or four things people get asked about constantly, a metric definition, a table schema, a runbook, and writing them as OKF concepts this week. You'll know almost immediately whether the format earns its place in your stack or whether your actual problem was volume all along, in which case RAG was the right call from the start.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article