I’ve built a self-improvement process into my AI operating system, AgenticOps. It collects evidence from real tasks, identifies recurring problems, and makes certain fixes automatically. Bigger changes still need my review.
The inspiration came from Harness², a research project from Google Research and Northwestern University. Its central idea is clever: an agent can change the system around itself, test those changes, and retain what helped.
That’s useful, but a production system needs limits. My agents handle real data and have connections that can publish content or send email. I want them to improve their setup. I don’t want them rewriting their own permission boundaries.
What an Agent Harness Actually Does
A harness is everything around the AI model. It includes instructions, tools, reusable skills, memory, safety rules, and the coordination that determines how work happens.
The same model can perform very differently inside different harnesses. Better instructions or a clearer process can improve results without changing the model itself.
Manually tuning that setup takes engineering time. Earlier approaches used AI to improve a harness against a fixed set of tasks before deployment. The difficulty appears when a new task doesn’t resemble those examples.
Harness² moves improvement into the task itself. That raises two problems:
- There’s no answer key. During live work, the system may not know exactly what a good result looks like.
- Improvements don’t always transfer. A change that helps one task might hurt another.
How the Two Harness² Loops Work
Harness² divides the harness into eight components: a system prompt, guardrails, memory, skills, tool guidance, scripts, subagents, and a middleware plugin.
These cover both instructions and behavior. Subagents delegate parts of a job. Scripts handle programmed actions. Middleware can influence how the agent’s operations run.

Loop One: Improve the Current Task
Every task begins with the same base harness. The agent runs the task, then uses that initial attempt as a reference. A limited retry budget allows it to test alternatives before returning the result.
The improvement cycle has three stages:
- Propose. Create two candidate harnesses. One addresses general issues, such as planning or tool use. The other targets knowledge or procedures specific to the job.
- Probe. Run both candidates on the same task with the same model. Compare the results with the reference run.
- Compose. Use those comparisons to build the next version. The system can merge useful changes, resolve conflicting edits, or leave the harness unchanged.
Each decision must connect to something the system observed. A convincing explanation alone isn’t enough.
These comparisons provide evidence, not proof. AI models can produce different results across repeated attempts. Still, testing a change offers a stronger basis for improvement than simply asking an agent to rewrite its instructions.
Loop Two: Retain Useful Experience
The second loop reviews the wider process. It identifies which changes helped, which problems they addressed, and the conditions where they worked.
Only lessons supported by run comparisons enter the library. Each saved entry contains a working harness, including its eight components, plus an experience.md file.
This goes beyond collecting reusable skills. A saved harness can contain different hooks, control flows, or guardrails.
Importantly, the next task doesn’t automatically inherit the previous task’s improved harness. It starts from the same base again. Its improvement loop can then consult the growing library and draw on relevant experience.
Four Lessons I Took From the Results
The researchers tested Harness² across benchmarks covering coding, office tasks, and legal work. The project site includes the benchmark results. Four points stood out to me:
- A better harness can outweigh a bigger model. In the reported tests, a smaller model with an improved harness could outperform a larger model using its base setup.
- Stronger models generally gained less. Smaller models showed the larger improvements.
- Validation drives the gains. The value comes from testing changes and comparing outcomes, not from self-editing alone.
- Basic process fixes matter. The biggest wins came from fixing how work happened, rather than making agents appear more intelligent.
What I Built Into AgenticOps
I started with evidence. My agent console tracks interactive and background threads, including whether they’re running, open, or complete.
At the end of a job, the agent checks whether the setup wasted time, forced a guess, or required a workaround. This includes work completed through Claude or Codex. It records a note only if another agent would likely encounter the same problem.
That filter matters. A successful task can hide a poor process. An agent might work around broken guidance and finish the job, while spending unnecessary time and tokens along the way.
Those reusable observations accumulate in a friction log.
A Weekly Review With Two Passes
Instead of reflecting after every task, my system runs a review each Monday morning. Batching the work reduces token consumption.
The review has two parts:
- Drift pass: Look for conflicting rules, bloated skills, broken references, and similar maintenance issues.
- Friction pass: Combine friction notes, failed jobs, and nonblocking code review findings. Group them by their underlying cause.
Every commit already gets reviewed by a different agent. If Claude Code makes a change, Codex reviews it, and vice versa. The drift pass adds another check across the wider setup.
In one weekly run, the system gathered 654 signals. It grouped them into at most three proposals, each requiring evidence and a specific change. Without enough support, it makes no change.

Where I Keep Human Approval
The weekly review can automatically make clear-cut fixes to specific types of files. That gives the system a limited ability to improve itself.
Code changes take a different route. The review creates tickets in my Software Factory backlog. I can move those tickets through the process, review the completed work, and send approved changes into the nightly build and release process.
This is the key difference between a research experiment and my production application. AgenticOps has real connections and real consequences. Its hard guardrails, permission boundaries, and safety policies require my approval.
Permission to propose an improvement isn’t permission to change every control. I want the system to surface problems and handle bounded maintenance. Sensitive changes must remain under human control.
The Next Step: Separate Learning From Execution
I’m considering more of the Harness² approach, but I’d separate normal work from experimentation. Execution happens throughout the day. Learning could happen at night or in an isolated development workspace.
Optimization also needs to justify its cost. Roughly 80% of my requests are one-off questions or ad hoc tasks. They don’t each need a dedicated improvement cycle.
Repeatable workflows, such as research or motion graphics generation, are better candidates. A task classifier could identify recurring patterns and decide whether further testing is worthwhile. That’s an idea I’m considering, rather than something I’ve already implemented.
A lighter example is Matt Pocock’s retro skill. It reviews real sessions for problems, following a similar principle to my friction log.
The principle I’m building around is straightforward: collect evidence from actual work, identify recurring causes, and test meaningful changes. Keep the ability to improve separate from the authority to rewrite the rules.

