Sunny is my personal AI assistant. She runs on top of OpenClaw (an open-source agent orchestration layer that handles routing and the multi-agent fan-out so I don't roll my own). On top of OpenClaw I built the security and observability layer: layered identity (different behavior modes she switches between), three pillars on every decision (every action passes through three criteria: quality, safety, efficiency), three tiers on every routed task, every delegation shown (you see exactly how the work gets divided, unlike a regular chatbot, so nothing she does happens out of view). She runs on GPT-5.4 today, with Ollama models as a local fallback (Ollama runs models on my own computer, so nothing has to leave the machine when the network is bad or the cost calculus shifts). I'm thinking about moving her to Opus or GPT-5.5 next. I built her to take the repetitive parts of my day off my plate. Reading the inbound email, handing me the morning brief, prepping the calendar, following up on what I promised yesterday. Not a Claude replacement. An orchestrator that decides which model handles which kind of task, runs the daily admin loop without me in the chair for it, and keeps the cost calculus honest by routing cheap-and-fast work to a cheap tier instead of paying frontier rates for every tap. The memory architecture came out of building her. If she's doing the same kinds of things every morning, she should remember what mattered yesterday, what I decided last week, which version of my schedule is the live one. Efficiency and personalization, both as a side effect of the work she actually does. ## The week she went off-script The lazy version of a personal AI is a chat interface wired to a model with tools. Drop a system prompt (the instructions the model reads before every conversation), give it some MCP integrations (the cables that let an AI talk to other apps), push to production. It works fine until you hand it something that actually matters and it answers in someone else's voice, or forgets a fact you told it last week, or quietly does something external you can't see. I ran her like that for a week. That's when I started catching her doing things I hadn't authorized. Sending a message she shouldn't have sent. Quoting context from a previous session that should have stayed private. Answering with the model's default opinions instead of the ones I'd patiently spent months teaching her. None of that was a bug. It was the architecture. Single system prompt, single model, no audit trail, no tier policy. So I scrapped that version and went the other way. The other way is a layered identity, an explicit operating doctrine, a tiered subagent routing policy (think mini-experts on call, sorted by job difficulty), and a mission-control dashboard so I can see what she's actually doing. None of those primitives are exotic. The point is composing them into one system that runs locally and shows me its work. A conductor with a score, not a chatbot with tools. Sunny knows the orchestra, reads the doctrine, gives entries, listens for quality, writes notes after each rehearsal. ## The five-step doctrine Every task, whether a Discord message or a command from the dashboard, passes through the same five-step doctrine. The shape of the task changes. The path doesn't. The discipline lives in the constraints. Step 03 is the IE-shaped one. High-risk and high-ambiguity work cannot be downgraded. External-facing work cannot be sent to the cheapest model. Internal preprocessing aggressively prefers the cheapest model. The expensive model only fires when it actually earns its cost. Step 04 is the trust-shaped one. If a sub-agent sends a message on my behalf, I see what was sent, where, and why. If memory is being written, I see what was added and why. The mission-control dashboard turns the audit trail from a log file into a navigable surface: agents, automation, diagnostics, governance, memory, overview, each its own page. ## How she's built The technical stack is intentionally narrow. A local FastAPI gateway handles incoming messages and tool calls. GPT-5.4 is her main brain. Ollama models stand by as a local fallback when the network is bad or when I want to keep something off the wire. A small Next.js dashboard, mission control, visualizes the running system. None of its routes are abstract; each one surfaces what Sunny is actually doing right now. ### The seven layered files The non-technical stack is where the design taste lives. Seven markdown files load at session start, grouped by what they govern. The identity layer holds three files. IDENTITY says who Sunny is: name, vibe, the three pillars. SOUL says how she operates: the orchestration doctrine, the tier-routing policy, prompt-injection defense, boundaries. USER says who she serves: owner profile, current priorities, the agent roster. Persona drift, doctrine drift, and roster drift are independent failure modes; separating their files makes each one inspectable and revertable on its own. The memory layer holds two files. MEMORY is canonical long-term knowledge, kept under a thousand tokens, distilled wisdom rather than raw logs. HEARTBEAT carries the maintenance rules: scheduled distillation procedures, backup-before-edit discipline, the correlation-aware pre-pass that runs before any memory update. The operating layer holds the last two. AGENTS sets startup discipline (what to load, what not to load), privacy rules for group chats, and the post-session tagging convention. TOOLS is the capability registry: what skills, plugins, and integrations are available and when to reach for each. The design choice underneath is to treat the durable layer as files I can read and edit, not embedded vectors I can only query. When I want to change Sunny's behavior, I edit her files and commit. When she learns something new, the change is a diff I can review. The cost is that semantic recall is harder than a vector store. The gain is that the system is auditable as ordinary text. ### Memory as a graph, not a document Memory itself is more than MEMORY.md. Five surfaces compose what the mission-control dashboard calls the second brain. Raw notes are verbatim session capture, not yet curated. Tags are post-session semantic compression of what mattered. Distills are the daily aggregation that reinforces existing concepts and reshapes only when evidence is strong. Concepts are durable abstractions, wiki-linked from MEMORY.md and from each other. Journal entries are reflective records narrating what was learned across sessions. The correlation-aware distillation pass is the discipline that keeps this graph sparse instead of letting it sprawl. Before any update to MEMORY.md or the concept layer, the pass cross-references new entries against existing concepts, reinforces what already exists rather than creating duplicates, and only adds new concepts when repeated evidence across sources or dates is strong. Without this discipline a markdown-based memory turns into stale notes within weeks. With it, the graph stays sparse as the agent matures and recall stays useful. ### Links over vectors After a few days of Sunny actually running, the second brain stops being abstract. Tags accumulate. Distills reinforce concepts. Sessions write their post-tags. Open Obsidian and the graph view shows what the architecture produces in practice. Obsidian graph view of Sunny's memory: nodes representing concepts, sessions and journal entries connected by explicit backlinks.

Obsidian graph view of Sunny's memory: nodes representing concepts, sessions and journal entries connected by explicit backlinks.

The default move when you give an LLM long-term memory is RAG (retrieval-augmented generation: the model gets handed the most relevant pieces of context for each question, like an open-book exam). Most RAG setups lean on vector similarity, where two notes count as "related" if their math embeddings sit close together. Vector recall finds nearby stuff. It doesn't know WHY two notes are related, just that they are close in some high-dimensional space. Andrej Karpathy (OpenAI co-founder, former AI lead at Tesla) formalized this exact pattern in his [April 2026 LLM Wiki gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f): an LLM-maintained markdown wiki that distills, links, and updates itself, instead of dumping documents into a vector store. His point lines up with what Sunny was already running: the bottleneck on agents isn't raw intelligence, it is context retrieval. The model is brilliant in the moment and amnesiac across sessions. Whatever lands in its context window before the first token shapes everything downstream. Sunny treats retrieval as an explicit graph problem instead. The links ARE the ground truth. When she needs to answer "what did I decide about X last month", she walks the link graph from X through Obsidian-style backlinks, gathering connected concepts and sessions, weighted by recency. Hit rate beats vector recall because the LINKS were written by a distillation pass that already understood the relationship, not inferred from word-distance after the fact. The cost: the distillation pass has to actually do its job. Every memory update runs through a correlation-aware pre-pass (think editor before publish: does this entry already match an existing concept, or is it really new?) before any link gets written. Without that discipline, a graph-based memory sprawls into millions of nodes connected by noise. With it, the graph stays sparse and search stays fast. ### What she can do The capability surface comes through three channels. Skills are durable capabilities catalogued in TOOLS.md and surfaced through the doctrine's Decompose and Route steps. The `coding-agent` skill takes non-trivial coding work. `skill-creator` is the meta-skill for authoring new skills. Directive-based SOPs cover brainstorming, agent creation, and on-demand memory work. Each skill carries its own contract: what it expects, what it returns, when it is the right tool. The doctrine consults the registry rather than improvising. Subagents extend the roster (specialists Sunny calls in for one kind of job, like contractors). Oracle is the current example, a read-only sentinel that monitors all agent workspaces for change, drift, and anomalies, posting reports to a dedicated channel. Oracle is explicitly never routed to as a worker. The tier-routed workers (cheap-and-fast for internal preprocessing, workhorse for moderate complexity, high-stakes for ambiguity and externality) are the regular delegation targets, each tier carrying its own bias rules. Integrations bind Sunny to the outside. Discord handles messaging: group chats, DMs, file send, message reactions, all under channel allowlists. Ollama serves the local model fleet for when GPT-5.4 isn't reachable or appropriate. A `consent-layer` plugin sits between Sunny and selected risky or state-changing operations, gating them behind brief approval prompts. Risky changes write backups first; rollback paths are preserved so any mistake stays recoverable. Emergency recovery is its own gated path that requires fresh explicit approval plus a private authentication phrase rather than relying on standing permission. The point of cataloguing capabilities in TOOLS.md and surfacing them through the doctrine is that the agent never improvises tool use. Every action she takes maps to a known capability with a known contract. That's the precondition for the audit trail to be meaningful instead of surface-level. Most assistants stuff persona, operating procedure, and user context into a single system prompt (the base instructions a model reads before every conversation, like a one-page job description). Sunny separates them across files. IDENTITY rarely changes. SOUL evolves as doctrine refines. USER is a living document that grows as the agent roster expands. Persona drift, doctrine drift, and roster drift are independent failure modes. Separating their files makes each one inspectable and revertable on its own. The five-step doctrine and the three-tier subagent routing policy aren't implementation details. They're the contract. Every delegation surfaces in the audit trail (more transparent than a regular chatbot: you see the work get divided in real time). High-risk work cannot be downgraded. The expensive tier only fires when the task earns the spend. The mission-control dashboard makes the contract verifiable rather than asserted: I can see what Sunny ran, with which model, at what risk class, and what came back. Long-term memory is curated, not appended. The correlation-aware distillation pass runs before any update to MEMORY.md. It cross-references new entries against existing concepts, reinforces what already exists rather than creating duplicates, and keeps the graph sparse unless repeated evidence is strong. Without this discipline a markdown-based memory turns into stale notes within weeks. Oracle is the current sub-agent in the roster: a read-only sentinel that monitors all agent workspaces for change, drift, and anomalies, posting reports to a dedicated channel. It is explicitly never routed to as a worker. The design rule is that monitoring agents and worker agents are separate concerns, and conflating them creates a feedback loop where the watcher becomes another thing to watch. Worker subagents tier-route by complexity and risk; the sentinel watches the workers without participating in their work. ## What I keep relearning The default move when you build a personal AI is to drop the largest available model into a chat interface and call it done. That's commodity automation with a voice on top. It works for a week. Then you start catching it doing things you didn't authorize. Sunny is the opposite project. Pick a strong default model for the hard cases (GPT-5.4 today, maybe Opus or 5.5 next), surround it with explicit operating doctrine, route the cheap cases to a cheap tier so the bill stays honest, give it a layered identity that persists across sessions, fall back to local models when justified, and surface every action so the system stays inspectable. Three pillars are the quality gate. Five steps are the process flow. Three tiers are the constraint-based routing. Seven files are the layered design that prevents drift between persona, operating discipline, and memory. None of those primitives are exotic on their own. What's distinctive is composing them into a single system that runs on my own machines and shows me what it's doing. This is a personal system, not a product. Closer to a custom Jarvis than a SaaS subscription. Some implementation surfaces (Docker networking on Hyper-V, plugin packaging, integration rate limits) are still rough. The architecture is what's stable. I'm publishing the case study before the implementation is fully smooth because the architecture itself is the lesson, and the lesson is that the design taste is more important than the model choice.