Why I Am Destroying My Beautiful Machine
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
--- Richard Sutton, The Bitter Lesson, March 2019
I have spent the past three years building a harness.
Not a physical one. A cognitive one: YAML schemas, CLAUDE.md conventions, specialized agents, session templates, memory taxonomies, orchestration protocols. It is intricate. It reflects real thought about real problems.
It is also wrong.
This post is the autopsy.
I. What the Bitter Lesson Actually Says
Sutton's 2019 essay is short. It is also devastating. He documents seventy years of AI research (chess, Go, speech recognition, computer vision) and finds one pattern:
Researchers build human knowledge into their systems. This helps in the short term. It is personally satisfying. But it plateaus. It inhibits further progress. Breakthroughs arrive only when researchers abandon their handcrafted expertise and embrace general methods that scale with computation.
The two methods that scale without ceiling: search and learning.
The second lesson, developed in Sutton's Alberta Plan and the subsequent Era of Experience work with David Silver, goes deeper:
Intelligence is continual learning from experience, not batch training on human data. The world is vastly more complex than the agent. Prior knowledge is a never-ending fool's errand. Build meta-methods that discover, not repositories of discoveries.
We want AI agents that can discover like we can, not which contain what we have discovered.
II. The Harness as a Beautiful Trap
I spent a career in adversarial systems analysis: malware classification, threat modeling, quantitative risk. The instinct that career builds is to specify tightly. Pin down the inputs. Constrain the state space. Verify the invariants. That instinct is correct for formal verification and provably correct code. Applied to AI workflow scaffolding, it produces something that looks rigorous but acts as a cage.
Let me show you what I built, and where each piece violates the lesson.
YAML Input Packets
Structured schemas defining agent behavior. Roles, inputs, outputs, step sequences. This is what chess researchers did when they encoded grandmaster evaluation functions into their programs. All of it became irrelevant the moment deep search scaled.
The YAML schema assumes I know the optimal structure for every task. As models improve, they need less structure, not more. I am encoding my current understanding as invariant law.
CLAUDE.md
Perhaps the purest violation. I codified how I think Claude should think: tone, formatting preferences, workflow rules.
"Building in how we think we think does not work in the long run."
That sentence is literally about CLAUDE.md. I am encoding interaction features the way early computer vision researchers encoded edge detectors and SIFT features. The model will learn better ones. My conventions are technical debt that compounds in the wrong direction.
Specialized Named Agents
Research Paper Deep Dive Analyzer. Slides Agent. OK-Computer agents with specific names and predetermined roles.
Each one embeds my theory of how paper analysis or slide generation should work. General models get better at all of these tasks simultaneously. The gap between "general model" and "specialized agent" closes with every release.
I built a zoo of bespoke tools instead of a general capability.
PHIAL Taxonomy
My nth-order attack classification system, rooted in work I started at NATO CCDCOE in 2009. Handcrafted domain knowledge. A manually constructed taxonomy that assumes I can pre-specify the structure of emergent threats.
It is my SIFT features: clever, useful now, on a timer. The taxonomy requires manual updates as threats evolve. Better: a general classification system that learns taxonomy from data, not from my design.
I will keep "PHIAL" as a brand name for client deliverables. Operationally, it dies.
Memory Instruction Schema
Explicit memory slots with categories, 50-slot limits, manual pruning rituals. I am manually engineering what the model should remember about me. The schema encodes my theory of self.
Better: the model learns what matters from natural interaction, the way human relationships work. No explicit save commands. No structured categories. Just conversation, and the model infers.
session_state.md Templates
Rigid log formats encoding my theory of what matters in a session. Artifacts, decisions, TODOs, predetermined sections.
I am doing the model's job: deciding what to remember and how to structure it. As models get better at long-context understanding, these templates become unnecessary intermediaries.
III. The Pattern
Every component follows the same arc. I understand my workflow, so I encode that understanding into structure. The structure works in the short term. Then models improve. The structure becomes a constraint, not an aid. I have to update the structure manually to keep up. I end up working for the harness, not the other way around.
Sutton:
"The human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation."
My harness is complicated. It reflects real understanding of my work, my pedagogy, my aesthetics. That is exactly why it will fail.
IV. What Compliance Looks Like
The Bitter Lesson does not say "never use structure." It says: structure should emerge from data and experience, not from human design.
Here is the architecture I am moving toward:
+-----------------------------------------+
| NATURAL LANGUAGE INTENT |
| (Prose description of what you need) |
+--------------------+--------------------+
|
+--------------------v--------------------+
| GENERAL REASONING ENGINE |
| (The frontier model, whatever it is) |
| |
| Read, synthesize, code, verify, |
| teach, classify, analyze. |
| All learned. None hardcoded. |
+--------------------+--------------------+
|
+--------------------v--------------------+
| RETRIEVABLE KNOWLEDGE CORPUS |
| (Everything you have done, searchable, |
| not pre-categorized) |
| |
| Past sessions, papers, code, |
| blog posts, client deliverables. |
+--------------------+--------------------+
|
+--------------------v--------------------+
| MINIMAL CONTEXT ANCHORS |
| (Identity + one meta-preference) |
| |
| Daniyel Bilar, Chokmah LLC. |
| Adversarial review preference. |
+-----------------------------------------+
No YAML. No specialized agents. No rigid templates. No handcrafted taxonomies.
Just: intent, general model, corpus, output.
The Translation
| What I Had | What I Am Building |
|---|---|
| YAML agent definitions | Natural language intent files |
| 50-slot memory schema | Experience-based inference from conversation |
| session_state.md templates | Free-form narrative + auto-generated summary |
| Specialized named agents | One general agent with context switching |
| PHIAL taxonomy | Retrieval-based dynamic classification |
| CLAUDE.md conventions | Shrinking to identity + one meta-preference |
| Lab facilitation scripts | General "facilitate learning" intent |
| Terraform configs | High-level intent files |
V. The Rituals
The Shrinking Convention Rule. Every month, I delete 20% of CLAUDE.md. If performance degrades, I add back only what was essential. The goal: the document approaches zero as model capability approaches infinity.
The Generalization Test. When I find myself building a new specialized agent or YAML schema, I ask: can I express this as natural language intent? If yes, I do it. If no, I ask whether the limitation is in the model or in my imagination.
The Meta-Harness Question. After every ten sessions, I ask the model: based on these transcripts, how should we change how we work together? Let the system propose workflow improvements. Test them. Keep what works. This is Sutton's meta-learning: the harness learns how to harness itself.
VI. What Survives
Not everything handcrafted is waste.
Client-facing frameworks (PHIAL). Brand value is real. The name stays. The ontology goes.
Formal verification (Coq, eBPF). Mathematical truth is not learned from data. Proofs are proofs.
Regulatory compliance. Rules that must be followed exactly, not interpreted.
Infrastructure safety. Until models can prove correctness, production Terraform stays human-reviewed.
The trend is clear anyway. The scope of "must be handcrafted" shrinks every quarter.
VII. The Brutal Truth
Sutton:
"The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach."
My harness is a favored, human-centric approach. It reflects years of careful thought about my work, my pedagogy, my aesthetics. The Bitter Lesson asks me to discard it, not because it is bad, but because general methods will soon make it obsolete, and the complexity I have built will prevent me from using them.
The upgrade is not about making my harness better. It is about making my harness disappear, replaced by natural language intent, general model capability, and searchable experience.
"We want AI agents that can discover like we can, not which contain what we have discovered."
Apply this to the harness itself. I want a harness that can discover how to work with me, not one that contains my current theory of how to work.
That is the upgrade. Everything else is implementation detail.