CKF Project Review: From CKF-0.1 to CKF Compiler v1.03.1
A scientific retrospective of the CKF Compiler, tracing the journey from CKF-0.1 (≈10% semantic preservation) to v1.03.1 — the first balanced release that simultaneously preserves meaning, structure, retrieval surface, sanitation, metadata and traceability.
Talk to this article
This post exists as a CKF package. Load it into your favorite LLM and discuss, summarize or apply its ideas.
CKF Project Review: From CKF-0.1 to CKF Compiler v1.03.1
Author: Paulo Tomazinho, PhD · CKF Research · May 2026
1. Project origin
The CKF project was born from a central observation about today's AI infrastructure:
Most of the knowledge consumed by AI agents is still stored in formats designed for humans, not for machine reasoning.
PDFs, Markdown pages, technical manuals, corporate policies, legal documents, guidelines, scientific articles and web pages organize knowledge through prose, sequence, headings, visual hierarchy, examples, emphasis, contrast and implicit relations.
AI agents, on the other hand, operate over: retrieval chunks, embeddings, context windows, schemas, tool calls, graphs, structured payloads, token limits, provenance and traceability.
This gap produced CKF's foundational hypothesis:
Structural inference should be shifted from inference time to compile time.
Instead of forcing the model to reconstruct hierarchies, exceptions, rules, scope, precedence and applicability from raw text on every answer, CKF proposes compiling human documents into structured, auditable, agent-consumable units.
From the start, CKF was never conceived as a new model, database, graph or protocol. It was conceived as a compiled intermediate representation for knowledge.
2. Conceptual foundation
One of the project's first accomplishments was to draw a precise line around what CKF is not: a vector store, a knowledge graph, a graph database, a formal ontology, a fine-tuning technique, a RAG architecture, a GraphRAG replacement, an MCP replacement, a PDF replacement, a prompt library or a system prompt format.
CKF was defined as:
a structured, versionable, auditable and traceable format for knowledge packages compiled from human documents and consumed by AI agents.
The goal was never to compete with vector databases, RAG, GraphRAG, MCP or PDF — it was to provide an intermediate layer that composes with all of them.
The most important formulation emerged early:
PDF preserves visual fidelity for humans; CKF preserves operational fidelity for machines.
3. CKF-0.1 — the initial failure
The first empirical compilation attempt produced what later became known as "CKF-0.1-style" output: technically structured, semantically shallow. It contained roughly 4 generic entities, 1 concept, 1 retrieval chunk, 1 atomic unit, superficial provenance, loss of argumentative structure and collapse into generic summarization.
Estimated semantic preservation: about 10%.
CKF-0.1 showed that syntactic structure is not knowledge preservation. Valid JSON is not enough; compact YAML is not enough; readable Markdown is not enough; a filled schema is not enough. The compiler had to become preservation-aware.
This initial failure created the need for semantic rubrics, structural analysis, strong provenance, source excerpts, confidence, source basis, section allocation, quality gates, regression tests and reproducibility packages.
4. Birth of the preservation methodology
The reference document chosen was "What CKF is not (and what it actually is)" — short but semantically dense. It contrasted CKF with vector databases, knowledge graphs, ontologies, fine-tuning, RAG, GraphRAG, MCP, PDF and prompt libraries; introduced CKF as an open format and intermediate representation (the LLVM IR analogy); and formulated operational fidelity, composition hallucinations and the pre-registered Study 2.
The methodology adopted a 20-unit substantive rubric, each unit scored 1.0 (preserved), 0.5 (partial) or 0.0 (absent).
Evaluation moved from a vague "Does the file look good?" to a scientific "Did the compiler preserve the substantive units of the source document?"
5. Structural evaluation
Beyond the semantic rubric, the project began measuring section population: entities, concepts, principles, heuristics, decision_rules, procedures, patterns, anti_patterns, causal_chains, contextual_triggers, if_then_rules, exceptions, mental_models, playbooks, qa_pairs, retrieval_chunks, atomic_units and source_traceability.
This count separated two phenomena:
- Semantic preservation — is the knowledge present?
- Structural allocation — is the knowledge in the right section?
6. CKF Compiler v1.0 — first big leap
CKF Compiler v1.0 extracted 15 entities, 14 concepts, 56 atomic units, 23 retrieval chunks and 112 source-traceability items. Semantic preservation jumped from ~10% to 97.5%.
v1.0 preserved contrastive structure, conceptual boundaries, distinctions between technologies, composition relations, operational claims, provenance and source excerpts. It was the first strong empirical result: CKF could preserve substantive knowledge when compilation was source-grounded and structurally oriented.
7. The schema allocation problem
Although v1.0 preserved nearly all content, it concentrated too much knowledge in atomic_units and retrieval_chunks. The richer sections (principles, decision_rules, procedures, anti_patterns, exceptions, playbooks, qa_pairs) stayed empty.
The compiler could preserve the sentence but did not always understand its operational role. For example:
"RAG is an architecture. CKF is a content format."
could surface as an atomic unit, but should also surface as a principle, a decision rule, or a contrastive claim.
The insight: faithful extraction and semantic allocation are different problems.
8. CKF Compiler v1.01 — promotion architecture
v1.01 introduced the first major architectural correction: the promotion system.
Atomic units and retrieval chunks should not be removed; they should be duplicated/promoted into rich sections when appropriate.
The project chose duplication over migration, preserving the retrieval surface while creating richer operational views.
v1.01 began populating principles, decision_rules, procedures, anti_patterns, exceptions, qa_pairs and mental_models. Total counted elements rose from 224 to 388 and semantic preservation hit 100%.
The schema-allocation problem was fixed without changing the CKF protocol — artifacts kept protocol_version: ckf-1.0, but the compiler deployment had evolved.
9. Section Allocation Quality
With v1.01 came the __saq (section allocation quality) metric, scoring whether knowledge was collapsed into atomic_units/retrieval_chunks or distributed across normative and operational sections.
CKF evaluation now had three layers: semantic preservation, structural population and operational allocation.
10. Issues exposed by v1.01
10.1 Language drift
Even in packages declared language: en, generated fields still appeared in Portuguese: labels, descriptions, QA questions and mixed-language principles.
10.2 Promotion artifacts
Some promoted items appeared truncated:
GraphRAG: Operational Layer vs.
PDF: Visual Fidelity vs.
Promotion alone was not enough — it required completeness filters, truncation detection, deduplication, quality gates and sanitation.
11. CKF Compiler v1.02 — quality instrumentation
v1.02 introduced/consolidated computable metrics: human_readability, ai_utility_score, section_allocation_score, language_fidelity and promotion_acceptance_rate. Quality stopped being a subjective judgment and became metadata.
v1.02 also started populating if_then_rules, playbooks, conditional structures, procedural structures and activation contexts better — moving CKF from merely retrievable to operationally actionable. It kept 100% semantic preservation and reached 393 counted elements.
Two problems persisted: partial language drift and surviving truncated artifacts. Diagnosis: filters were being applied mostly to new promotions, not to the whole package.
12. CKF Compiler v1.03 — global sanitizer
v1.03 introduced an important architectural shift: sanitation must run globally after merge and promotion.
reduce()
→ promoteAtomicsAndChunks()
→ sanitizeMergedPackage()
→ computeQuality()
→ serializeMarkdown()
v1.03 added the global package sanitizer, language lock, short-string language detection, truncation filtering, sanitizer audit and post-reduce cleanup. It eliminated residual language drift and truncated artifacts.
13. The v1.03 regression — over-sanitization
The v1.03 global sanitizer was right in intent but too aggressive: it applied isCompleteProposition() to fields like retrieval_chunks.title, procedures.name, playbooks.name, anti_patterns.name, concepts.label, entities.name and atomic_units.tags.
Result: retrieval_chunks dropped to 0; procedures to 0; if_then_rules to 0; playbooks to 0.
Fundamental lesson: a global sanitizer is necessary, but it must be field-aware. Titles, labels, names, tags, statements and source excerpts are not the same kind of text.
14. CKF Compiler v1.03.1 — field-aware sanitizer
v1.03.1 fixed the regression by making the sanitizer field-aware. Field roles now: label, title, name, tag, substantive, source_excerpt.
Apply propositional completeness only to substantive fields.
Fields like retrieval_chunks.title, procedures.name, playbooks.name, concepts.label, entities.name and atomic_units.tags are not required to be complete propositions.
15. Preserving retrieval chunks
v1.03.1 added a specific rule for retrieval_chunks: a chunk must not be removed just because its title is short or imperfect, as long as the body is valid (valid compressed_knowledge, valid or empty standalone_context, with source_refs and source_excerpts). If the title is bad, the compiler repairs the title — it does not delete the chunk.
In v1.03.1, retrieval_chunks went from 0 back to 18.
16. Rebuilding source traceability
v1.03.1 added a rebuild step to prevent orphan entries:
reduce()
→ promoteAtomicsAndChunks()
→ sanitizeMergedPackage()
→ ensureIds()
→ rebuildSourceTraceability()
→ computeQuality()
→ serializeMarkdown()
17. v1.03.1 results
v1.03.1 is the most balanced version so far: 100% semantic preservation, 511 counted CKF elements, 18 retrieval chunks, 66 atomic units, 46 principles, 6 decision rules, 4 procedures, 25 anti-patterns, 15 exceptions, 3 playbooks, 36 QA pairs, 261 source-traceability items, __saq = 1, human_readability = 0.8, ai_utility_score = 1.
v1.03.1 is the first balanced version of the CKF Compiler: it simultaneously preserves semantics, structure, retrieval surface, sanitation, metadata and traceability.
18. Quantitative evolution
| Criterion | v1.0 | v1.01 | v1.02 | v1.03 | v1.03.1 |
|---|---|---|---|---|---|
| Semantic preservation | 97.5% | 100% | 100% | 97.5% | 100% |
| Total counted | 224 | 388 | 393 | 393 | 511 |
| Entities | 15 | 13 | 15 | 17 | 12 |
| Concepts | 14 | 16 | 16 | 17 | 17 |
| Principles | 0 | 25 | 20 | 42 | 46 |
| Decision rules | 0 | 9 | 8 | 4 | 6 |
| Procedures | 0 | 4 | 5 | 0 | 4 |
| Anti-patterns | 0 | 8 | 11 | 42 | 25 |
| Exceptions | 0 | 5 | 5 | 15 | 15 |
| If-then rules | 1 | 0 | 2 | 0 | 0 |
| Playbooks | 0 | 0 | 3 | 0 | 3 |
| QA pairs | 0 | 27 | 27 | 37 | 36 |
| Retrieval chunks | 23 | 21 | 23 | 0 | 18 |
| Atomic units | 56 | 64 | 61 | 39 | 66 |
| Source traceability | 112 | 194 | 194 | 256 | 261 |
| Language drift | Yes | Yes | Partial | Fixed | Fixed |
| Truncated artifacts | Yes | Yes | Partial | Fixed | Fixed |
| Over-sanitization | No | No | No | Yes | Fixed |
19. Scientific evolution of the project
| Phase | Central question |
|---|---|
| CKF-0.1 | Can knowledge be serialized? |
| v1.0 | Can knowledge be preserved? |
| v1.01 | Can knowledge be structurally allocated? |
| v1.02 | Can quality be instrumented? |
| v1.03 | Can package integrity be globally sanitized? |
| v1.03.1 | Can global sanitation be precise and preserve retrieval readiness? |
20. Methodology evolution
The methodology now includes a semantic rubric, structural counting, provenance analysis, section-allocation analysis, language fidelity, artifact sanitation, traceability consistency, metadata calibration, regression tests, code audits, reproducibility packages and longitudinal comparison.
The project stopped asking "Does CKF work?" and started asking:
Which compiler dimension improved, which regressed, and which mechanism explains the result?
21. Main conceptual contributions
21.1 Operational fidelity
PDF answers: how should this document look to a human? CKF answers: how should this knowledge operate inside an agent?
21.2 Compile-time structural inference
Structure should be inferred once, at compile time, not reconstructed repeatedly at inference time.
21.3 Composition hallucinations
The project refined "hallucination" into a specific class: relational-reconstruction failures under retrieval pressure.
21.4 CKF as IR
CKF is not a runtime, model, database, protocol, graph or prompt. CKF is an intermediate representation between human documents and agentic execution systems.
22. Current project state
With v1.03.1, the CKF project has a source-grounded compiler, structured schema, item-level provenance, semantic chunking, stable span IDs, a reducer, a promotion architecture, quality gates, metadata calibration, language lock, a global package sanitizer, field-aware sanitation, source-traceability rebuild, regression tests, reproducibility packages, a scientific paper and a longitudinal evaluation methodology.
The project evolved from shallow structured summarization to preservation-aware, auditable, operationally useful knowledge compilation.
23. The most important historical transition
The most important transition was moving from valid structured output to scientifically evaluated semantic preservation. The second was moving from semantic preservation to preservation + section allocation + retrieval readiness + sanitation + traceability consistency. That second transition is what makes v1.03.1 a milestone.
24. Current limitations
v1.03.1 is the most balanced version but not the final one:
if_then_rulesstill stays at 0 in the evaluated artifact;- some language-mismatch false positives can still occur;
- short decisions like "Use CKF." may still be removed by aggressive rules;
source_titleinference may still pick an internal heading over the main title;- evaluation still concentrates on one canonical document;
- downstream benchmarks comparing agents answering questions over PDF vs CKF are missing.
25. Next research trajectory
The natural next phase is moving from preservation to downstream use: PDF vs CKF on retrieval, traceability and composition-failure tasks. Compare raw PDF, CKF JSON, CKF Markdown and CKF YAML across factual, contrastive, multi-hop, operational, exception-laden and provenance-related questions. Metrics: factual accuracy, completeness, traceability, hallucination rate, composition correctness, token cost and answer efficiency.
This study would close the narrative arc:
- Does CKF preserve knowledge?
- Does the compiler explain that preservation?
- Do agents actually use that knowledge better?
26. Final historical assessment
CKF evolved through an unusual and scientifically healthy sequence: conceptual proposal, initial empirical failure, methodology creation, preservation leap, schema-allocation discovery, promotion architecture, quality instrumentation, global sanitizer, over-sanitization regression, field-aware fix, reproducibility, scientific formulation.
The main achievement of CKF up to v1.03.1 is not just having created a structured format. It is having established a preservation-aware compilation methodology for agent-consumable knowledge.
v1.03.1 is the project's first mature equilibrium: a compiler that preserves meaning, allocates structure, maintains retrieval readiness, fixes language drift, removes truncated artifacts, calibrates metadata and rebuilds traceability. It is the most important technical and scientific milestone of CKF to date.