Why CKF exists
For most of human history, knowledge has been encoded for human readers. Books, papers, manuals, slides, contracts, clinical guidelines — all designed for eyes, attention, and inference performed by people. These formats served their purpose well. They still do.
But a new class of reader has appeared.
Language model agents now read, retrieve, summarize, reason about, and act on the contents of human documents. The technology to do so has improved faster than the format of the documents being read. Today, agents consuming PDFs, DOCX files and Markdown notes are reading artifacts designed for an entirely different cognitive substrate. The mismatch produces specific, observable failures.
When a language model receives a PDF as raw text, it must reconstruct — in inference time and from prose alone — every relation a human reader would have inferred from layout, context, and prior knowledge. Hierarchies between rules and exceptions are implicit. Procedural dependencies are buried in narrative. The scope of one section conditions another section nine pages away. The model rebuilds this structure probabilistically, from fragments, on every query. The rebuild is the source of many errors that the literature calls "hallucinations" but that are more accurately described as failures of composition: outputs that contradict information present in the context, not because the model lacked the information, but because the relations between fragments were never made explicit.
The Compiled Knowledge Format proposes that part of this structural inference can be moved earlier.
Instead of asking the agent to rediscover the rules, exceptions, procedures, and provenance of a document each time it is read, those structures can be extracted once, encoded explicitly, and served in a typed format that agents read natively. The original document does not disappear — the PDF remains, available for human reading. The compiled package is a parallel artifact: same source, different audience.
This is the architectural intuition behind CKF. A compile-time pass over human documents that produces an intermediate representation optimized not for human eyes but for machine consumption. The analogy is to compilers in software engineering: source code is rich, narrative, and human-readable, but it is not executed directly. Compilers transform it into intermediate representations that runtimes can consume efficiently. Applied to knowledge, the same principle suggests that documents authored for humans need not be the artifact that agents reason over.
CKF is not a replacement for retrieval-augmented generation, graph-based RAG, vector databases, the Model Context Protocol, or knowledge graphs. It is a schema for what those systems carry when their content is structured knowledge rather than unstructured text. A .ckf package can be indexed by a vector database, traversed by a graph traversal algorithm, or served as a native MCP resource. The format composes with the existing stack; it does not compete with it.
Two commitments shape the design.
The first is honesty about provenance. Every claim, rule, exception, and procedure in a CKF package carries a source basis label and a confidence score, with a span-level link back to the original document. An agent answering from a CKF package can be audited: each output can be traced to a specific passage in a specific source. This matters because agents in regulated domains will eventually be required to justify their decisions. Provenance cannot be retrofitted onto unstructured chunks after the fact.
The second is honesty about evidence. Whether CKF improves grounded retrieval enough to justify its compilation cost is an empirical question, not a settled one. The first pilot, run with ten questions and a single model family for both agent and judge, produced near-ceiling scores that the benchmark could not differentiate. A pre-registered confirmatory study is in preparation to test the hypothesis under retrieval pressure: smaller context budgets, lower top-k retrieval, multi-hop questions, an independent judge model, and a dedicated benchmark — COMPGAP — built specifically to isolate composition failures from other failure modes.
If the confirmatory study supports the hypothesis, CKF will have demonstrated that structural compilation reduces a measurable class of errors at acceptable cost. If it does not, CKF will still have contributed a vocabulary — composition hallucination, schema-stable retrieval, compiled knowledge — that the field can use independently of any decision about the format itself.
This project is published under MIT license. The reference implementation, the format specification, the benchmark, and the experimental data are openly available. The research is conducted in public.
CKF exists as an attempt to bridge a gap that the current AI stack improvises around, case by case. It does not claim to be inevitable, nor universal. It is one proposal for what the layer between human documents and machine readers could look like — made falsifiable by the experimental work that follows.
The work continues. The hypothesis remains under test.
From narrative to structure. From documents to cognition. From content to context.
CKF v1.0 for this page has not been compiled yet. Downloads become available once an admin runs the compiler.