M-Clone

Every bank now ships an assistant, and every one of them phones home. This project rebuilds the whole surface — accounts, credit cards, budgets, goals, Plaid-linked real data — and then answers the interesting question: how good can a money assistant be if it never leaves the phone? Chat history, financial ledgers, budgets, and Plaid tokens live on-device in SwiftData and the Keychain. There is no backend and no server-side database; the only hosted piece is a thin Plaid OAuth/webhook relay.

The assistant is a hybrid: a deterministic router with typed financial tools and calculators does the load-bearing work, and Apple's on-device foundation model assists where language helps — with the model's authority capped by construction. On the live corpus, the current arm routes 86.5% of prompts to the right capability family and has leaked zero out-of-scope requests in 35 attempts.

The deeper deliverable is the discipline around that number. The router's judgment is governed by a real eval product — versioned corpus packs, preregistered experiments, a failure taxonomy, a cryptographically sealed holdout — and by commit gates that keep every test model-free. The claim isn't "the assistant feels smart." It's "here is the corpus, here is the score, here is the evidence."

In July the suite itself went under review, scored against an outside eight-part standard by a reader with no session history — and the verdict shipped before any of it was repaired: two parts met, five partial, one missing.

Two host shells share one Swift package; everything that matters lives in the package. The assistant's spine is deliberately boring: the model proposes an intent, deterministic routing decides what runs, and typed tools compute answers from the real ledgers — the model never free-generates a balance.

The assistant's quality is not a vibe; it's a shipped artifact with its own versioning, its own guards, and its own report. Corpus packs cover development, confusables, workflows, journeys, and over-refusal probes. Experiments are preregistered before they run. Failures are classified in a written taxonomy. And an 81-case holdout sits sealed behind a hashing workflow so no tuning session can ever quietly train on the exam.

Every commit pays a fast toll; the full routing gate runs the expensive machinery:

git commit → .claude/gate.sh (<120s, every commit)pytest scripts/tests · corpus validation · no-remote-LLM guard · model-free guard · anti-cheatthe full assistant-routing gateXCTest on an iPhone simulator · capability-pack native scoring · det-harness smokenightly 06:17 — both run again across the fleet; the digest lands before I wake

The same discipline covers the insights engine — the feed that turns ledgers into observations. Thirty-five engineered fixtures pin its rules; live narration runs are scored per-row, on device, and adjudicated by actually reading every output rather than trusting the metric. That habit exists because a metric lied once, and the story of how is below.

Grading the grader#

A suite this invested in measurement eventually has to measure itself. A reader with no session history scored it against an outside eight-part standard for eval work and filed the verdicts with the failing lines quoted — then the repair was deliberately left for separate, reviewed work. Nothing in the corpus, the gates, the thresholds or the grading code moved. This entry says audited, not fixed, and the eight verdicts are in the deep dive above.

The load-bearing finding is the one worth carrying here: the nightly check asserts that 61 cases were scored, not that they were answered correctly, so quality could fall through the floor and the run would still print a pass. And the matcher behind the headline routing numbers is documented in the repo as too generous — it accepts fields the answer key never asked for. Every score on this page, sealed ones included, is an upper bound until that ruler tightens.

1. NX-01Teach the nightly to fail on wrong answers, not just missing ones — and to report a broken machine differently from a broken answer.
2. NX-02Tighten the matcher, put the sealed holdout on a read budget, and let no agent grade a change it wrote. Scores are expected to move down; that's the ruler getting honest, not a regression.
3. NX-03Cover the safety class that isn't covered: personal data arriving in the question, with benign near-misses alongside it so the fix can't just be a more nervous assistant.
4. NX-04Template-first rendering on hardware tiers where the on-device context window stays small.
5. NX-05Deeper insight narration under the same per-row adjudication bar that caught the 1.000 lie.
6. NX-06Grow the capability corpus from real sittings — every confused routing becomes a case before its fix.
7. NX-07Keep the relay boring: OAuth return, AASA, webhook-to-push — nothing else grows server-side.