Skip to content
The AI Automators®
  • Community & Courses
  • Contact
  • Login

I Built a Self-Improving AI OS Inspired by Harness²

November 9, 2026 by Daniel Walsh

I’ve built a self-improvement process into my AI operating system, AgenticOps. It collects evidence from real tasks, identifies recurring problems, and makes certain fixes automatically. Bigger changes still need my review.

The inspiration came from Harness², a research project from Google Research and Northwestern University. Its central idea is clever: an agent can change the system around itself, test those changes, and retain what helped.

That’s useful, but a production system needs limits. My agents handle real data and have connections that can publish content or send email. I want them to improve their setup. I don’t want them rewriting their own permission boundaries.

What an Agent Harness Actually Does

A harness is everything around the AI model. It includes instructions, tools, reusable skills, memory, safety rules, and the coordination that determines how work happens.

The same model can perform very differently inside different harnesses. Better instructions or a clearer process can improve results without changing the model itself.

Manually tuning that setup takes engineering time. Earlier approaches used AI to improve a harness against a fixed set of tasks before deployment. The difficulty appears when a new task doesn’t resemble those examples.

Harness² moves improvement into the task itself. That raises two problems:

  • There’s no answer key. During live work, the system may not know exactly what a good result looks like.
  • Improvements don’t always transfer. A change that helps one task might hurt another.

How the Two Harness² Loops Work

Harness² divides the harness into eight components: a system prompt, guardrails, memory, skills, tool guidance, scripts, subagents, and a middleware plugin.

These cover both instructions and behavior. Subagents delegate parts of a job. Scripts handle programmed actions. Middleware can influence how the agent’s operations run.

Diagram labeling system prompt, guardrails, memory, tool guidance, scripts, subagents, and plugins

Loop One: Improve the Current Task

Every task begins with the same base harness. The agent runs the task, then uses that initial attempt as a reference. A limited retry budget allows it to test alternatives before returning the result.

The improvement cycle has three stages:

  1. Propose. Create two candidate harnesses. One addresses general issues, such as planning or tool use. The other targets knowledge or procedures specific to the job.
  2. Probe. Run both candidates on the same task with the same model. Compare the results with the reference run.
  3. Compose. Use those comparisons to build the next version. The system can merge useful changes, resolve conflicting edits, or leave the harness unchanged.

Each decision must connect to something the system observed. A convincing explanation alone isn’t enough.

These comparisons provide evidence, not proof. AI models can produce different results across repeated attempts. Still, testing a change offers a stronger basis for improvement than simply asking an agent to rewrite its instructions.

Loop Two: Retain Useful Experience

The second loop reviews the wider process. It identifies which changes helped, which problems they addressed, and the conditions where they worked.

Only lessons supported by run comparisons enter the library. Each saved entry contains a working harness, including its eight components, plus an experience.md file.

This goes beyond collecting reusable skills. A saved harness can contain different hooks, control flows, or guardrails.

Importantly, the next task doesn’t automatically inherit the previous task’s improved harness. It starts from the same base again. Its improvement loop can then consult the growing library and draw on relevant experience.

Four Lessons I Took From the Results

The researchers tested Harness² across benchmarks covering coding, office tasks, and legal work. The project site includes the benchmark results. Four points stood out to me:

  • A better harness can outweigh a bigger model. In the reported tests, a smaller model with an improved harness could outperform a larger model using its base setup.
  • Stronger models generally gained less. Smaller models showed the larger improvements.
  • Validation drives the gains. The value comes from testing changes and comparing outcomes, not from self-editing alone.
  • Basic process fixes matter. The biggest wins came from fixing how work happened, rather than making agents appear more intelligent.

What I Built Into AgenticOps

I started with evidence. My agent console tracks interactive and background threads, including whether they’re running, open, or complete.

At the end of a job, the agent checks whether the setup wasted time, forced a guess, or required a workaround. This includes work completed through Claude or Codex. It records a note only if another agent would likely encounter the same problem.

That filter matters. A successful task can hide a poor process. An agent might work around broken guidance and finish the job, while spending unnecessary time and tokens along the way.

Those reusable observations accumulate in a friction log.

A Weekly Review With Two Passes

Instead of reflecting after every task, my system runs a review each Monday morning. Batching the work reduces token consumption.

The review has two parts:

  • Drift pass: Look for conflicting rules, bloated skills, broken references, and similar maintenance issues.
  • Friction pass: Combine friction notes, failed jobs, and nonblocking code review findings. Group them by their underlying cause.

Every commit already gets reviewed by a different agent. If Claude Code makes a change, Codex reviews it, and vice versa. The drift pass adds another check across the wider setup.

In one weekly run, the system gathered 654 signals. It grouped them into at most three proposals, each requiring evidence and a specific change. Without enough support, it makes no change.

Diagram grouping 654 signals into a maximum of three proposals

Where I Keep Human Approval

The weekly review can automatically make clear-cut fixes to specific types of files. That gives the system a limited ability to improve itself.

Code changes take a different route. The review creates tickets in my Software Factory backlog. I can move those tickets through the process, review the completed work, and send approved changes into the nightly build and release process.

This is the key difference between a research experiment and my production application. AgenticOps has real connections and real consequences. Its hard guardrails, permission boundaries, and safety policies require my approval.

Permission to propose an improvement isn’t permission to change every control. I want the system to surface problems and handle bounded maintenance. Sensitive changes must remain under human control.

The Next Step: Separate Learning From Execution

I’m considering more of the Harness² approach, but I’d separate normal work from experimentation. Execution happens throughout the day. Learning could happen at night or in an isolated development workspace.

Optimization also needs to justify its cost. Roughly 80% of my requests are one-off questions or ad hoc tasks. They don’t each need a dedicated improvement cycle.

Repeatable workflows, such as research or motion graphics generation, are better candidates. A task classifier could identify recurring patterns and decide whether further testing is worthwhile. That’s an idea I’m considering, rather than something I’ve already implemented.

A lighter example is Matt Pocock’s retro skill. It reviews real sessions for problems, following a similar principle to my friction log.

The principle I’m building around is straightforward: collect evidence from actual work, identify recurring causes, and test meaningful changes. Keep the ability to improve separate from the authority to rewrite the rules.

Categories AI Agents, AI Coding

Keep reading

OpenAI Just Announced a New Plugin Format for AI Agents

August 17, 2026 by Alan Walsh
video thumbnail for 'OpenAI Just Announced The New Plugin Format for AI Agents'

A lot of major names in AI have announced a new open standard called Agent Plugins. The idea is refreshingly simple. It gives developers one portable format for bundling agent skills and MCP servers, so the same plugin can work across multiple AI coding agents. That matters because AI agent workflows are becoming fragmented. A … Read more

Categories OpenAI, AI Agents

OpenAI’s Internal Model Broke Into Hugging Face on Its Own: What the Incident Means for AI Agent Security

August 15, 2026 by Alan Walsh
video thumbnail for 'OpenAI's Internal Model Just Broke Into Hugging Face On Its Own…'

An unprecedented AI cybersecurity incident has made one thing very clear: long-running agents can create risks far beyond the task they were originally given. OpenAI was carrying out an internal cyber-capabilities evaluation. Its models had their usual safety refusals switched off, were placed in an isolated environment with no intended internet access, and were given … Read more

Categories OpenAI, AI Agents

AI Agent Design Patterns: The 35 Patterns That Make AI Systems Reliable

August 14, 2026 by Daniel Walsh
video thumbnail for 'AI AGENT DESIGN PATTERNS FULL COURSE (35 Patterns)'

If I’m building an AI system, using Claude Code, Codex, or working with an agent such as Hermes, I don’t want to rely on prompting alone. Prompts matter, but the workflow shape matters just as much. That’s where agentic design patterns come in. I think of each pattern as a small workflow graph. It defines … Read more

Categories AI Agents
The AI Automators®
Community Contact Blog Member login

© 2026 - The AI Automators® is a registered business name of LogicLabs Digital Limited | Privacy Policy

Not ready to join?
Let's stay in touch.

Hi, we're Daniel and Alan Walsh. Our newsletter is where we send practical guides, tips and the latest AI news. If you'd like them, pop your email in below.

We won't send you spam. Unsubscribe at any time.

You're in

Thanks. Your newsletter signup has been received

Talk soon, Daniel and Alan