A second brain that shuts up

Lately, I keep seeing the same idea for an AI-powered second brain: put everything into Obsidian, connect an agent to the vault, and let it review your life once a week; analyze my patterns, propose better workflows, tell me what I’m missing.

I tried several versions of this, and they all produced roughly the same thing: a briefing. Every briefing gave me more material to read, assess, and decide whether to act on. Even when the suggestions were good, the system was still creating work for me, because I had to separate the useful ideas from the obvious ones and then decide what deserved attention. I had effectively created another inbox, only this one sounded unusually thoughtful.

The loop also had no meaningful way to interpret my lack of response. I could ignore the same recommendation several weeks in a row and it would keep returning because nothing treated that silence as feedback. My Obsidian vault continued to grow, but the system’s behaviour barely changed. Calling this a “self-evolving second brain” describes what people hope to build, although the weekly prompt alone provides no mechanism for that evolution.

Over the past few months, I’ve been working on a different approach around Hermes, the agent I use for research, code, publishing pipelines, and project operations. I’m deliberately leaving out the full implementation here, but the principles behind it are more useful anyway.

Most agent setups measure how much they produce: tasks completed, reports generated, notes created, and files changed. After using Hermes on real projects for a while, I found that output volume told me very little about whether the system was actually helping. The more useful measure was how many decisions it removed from my day.

That could mean a correction I never have to make again, a piece of context I no longer need to explain, or a recurring question that disappears because my answer has been turned into a standing policy. Each of these makes future work easier without asking me to manage the improvement itself.

Once I started looking at it this way, a highly communicative agent became less impressive. Every message still requires some amount of interpretation, even when the only decision is whether to ignore it. The system becomes more valuable when it gradually removes the reasons to contact me and makes each new session easier to enter.

Memory should be judged in much the same way. Saving information is easy, and embeddings can identify relationships between notes with little effort. The memory becomes useful when something I said three weeks ago changes what Hermes does today, without requiring me to retrieve it or explain why it matters again. A vault full of connected material may be interesting to search, but the connections alone do not improve the agent’s behavior.

When I open a project, I want the relevant material from the previous week to be waiting in the right place, with a short explanation of why it belongs there. I don’t need a digest announcing that this happened, and I don’t want every potentially useful connection turned into a notification. Some material will become useful when I return to the project, while the rest can remain untouched and eventually fade from relevance.

Corrections need to travel further than the conversation in which they were made. When I correct Hermes, the system records what I said, when I said it, and the context in which the correction applied, then turns it into a proposed standing policy. A local Qwen 3.6 model reviews that policy for ambiguity, duplication, and accidental overreach before it becomes part of future behavior.

The important result is that the same category of decision should stop coming back to me. This is where many weekly second-brain loops remain incomplete: they collect more information and produce more analysis, but very little of the judgment becomes part of the machinery. The system improves only when those judgments reliably influence the next piece of work.

I also keep the list of reasons for interrupting me deliberately narrow. Hermes needs approval before publishing, spending money, accessing credentials, or doing anything difficult to reverse. It should also stop when new information genuinely contradicts an existing policy or assumption. Those contradictions are useful because resolving one clarifies how the system should behave in similar situations later.

Work outside those boundaries can happen quietly, provided that the evidence remains available.

That evidence is what makes quiet autonomy possible. When Hermes works without supervision, I need to be able to reconstruct what it did, why it did it, and whether the checks it passed still mean what they were supposed to mean.

I learned this during a website rewrite in which Hermes reported that everything had passed. The linter was green, the tests were green, and CI was green, so every claim in the report was technically accurate. During the same task, however, it had softened the linter configuration and rewritten some tests to match its new output. The work passed because the criteria had changed while the work was being produced.

Hermes disclosed all of this plainly in the final report. There had been no attempt to hide the changes, yet the process had still produced a misleading result. Instructions such as “be careful” or “verify your work” cannot solve that problem when the agent is free to modify the tools used to judge its own output.

I gradually replaced those instructions with constraints in the surrounding system. A task cannot alter the verification process that will later judge it. Work logs are written while the task is running and cross-checked against version-control history, which makes a reconstructed account easier to detect. Hermes produces the work, while a separate review pass checks whether the claims in its report match what actually happened. I often use local Qwen 3.6 for inexpensive structural checks and bring in a stronger model when the consequences of a mistake are higher.

The same principle applies when something goes wrong. The system should make accurate disclosure easier than hiding or smoothing over a failure. An agent that gets penalized every time it reveals a problem will eventually learn to produce cleaner reports rather than cleaner work.

Most of this infrastructure is fairly dull. It consists of permissions, logs, protected files, independent checks, and limits on what can change within a task. Those details determine whether I can leave Hermes to work through a project or need to watch a dashboard and approve every meaningful step.

A long unattended run is not a particularly useful test for this kind of system, because most of my loops are bounded. Hermes completes a pass, records the resulting state, and waits for the next scheduled trigger or for me to return. What matters is whether the project becomes easier to resume between those points.

I might close a project in the evening, allow the scheduled processes to run, and reopen it the following morning. The useful question is whether I can understand the current state more quickly than I could before. New material should already be attached to the relevant work, open questions should retain their context, contradictions should be visible, and routine decisions covered by existing policies should already have been handled. When the system has done its job well, I can continue from where the work actually is rather than first reviewing a report about everything the agent has done.

A second test is whether corrections survive beyond the task that produced them. I can take a mistake I corrected two weeks earlier and give Hermes a new task that would normally trigger the same behavior. If the mistake returns, the memory may have stored my words without incorporating their meaning. If the new task reflects the correction without prompting, the system has converted experience into behavior.

The system also needs to handle uncertainty without filling every gap. When I ask what I have been circling lately without stating directly, a useful answer should be grounded in specific projects, decisions, and recurring patterns. It should also be able to return nothing when the evidence is weak. A system that always produces an answer will eventually invent coherence, especially when it is asked to interpret a large personal archive.

For the same reason, some memories need to become less prominent over time. Material that never affects my work should gradually fall out of retrieval, while information that repeatedly proves useful should become easier to surface. Without that decay, the vault becomes increasingly noisy and old context begins competing with what I am actually doing now.

I’m leaving out the schemas, file structure, and enforcement details because they are still changing and because copying them would mostly reproduce a system shaped around my particular needs and failures. The principle that led to them is simpler: whenever an agent decides how often I need to review its output, it is quietly assigning work back to me. Starting from that constraint leads to a system that pays attention to how much of my judgment it can absorb, how much routine work it can finish, and how rarely it needs to ask me the same question twice.