Local models, frontier judgement — building a practical AI delivery workflow

As I used AI for more software work, I wanted to spend the strongest models’ capacity where it mattered most. Large logs, repetitive edits and first-pass transcriptions were consuming the same allowance I needed for architecture, debugging and review. I also had a MacBook Pro with an M5 Max and 128 GB of unified memory capable of running substantial models locally.

The question was practical: which work could those models take on, and how would I know the result was usable?

The result is a local delivery toolchain used alongside Claude and GPT-based coding workflows. It supports summarisation and generation through MCP, and a coding agent that edits files and runs checks in an isolated copy of a repository. The experiments shaped both the choice of models and the rules for using them.

Measure completed work, not just tokens per second

The first comparisons quickly challenged a simple assumption: the model that generates tokens fastest will finish the task first.

In three small September 1 probes, Qwen’s 35B model generated roughly twice as many tokens per second as GPT-OSS 120B. GPT-OSS nevertheless finished each task sooner.

Task	Qwen 35B generation	Qwen elapsed	GPT-OSS 120B generation	GPT-OSS elapsed
Simple logic question	122 tokens/s	20.0 seconds	77 tokens/s	1.1 seconds
Small coding function	144 tokens/s	7.3 seconds	77 tokens/s	1.5 seconds
Structured extraction	151 tokens/s	6.8 seconds	77 tokens/s	2.4 seconds

These were individual local probes, not a general model ranking. Qwen spent more of its output on reasoning. On a separate bug-finding task, it generated at 109 tokens per second but took 47 seconds; GPT-OSS generated at 55 and finished in 18 seconds. Under a tighter output budget, Qwen had exhausted the budget before returning a final answer.

That distinction changed the selection process. Generation speed, time spent reading the prompt, reasoning length and the usefulness of the final answer all mattered.

Different models earned different jobs

The setup grew into several roles rather than one default model for everything.

Model	Size in this setup	Useful role and observed limits
GPT-OSS 120B	About 65 GB of downloaded weights	General generation; shorter reasoning made it quicker on several early tasks despite slower token generation.
Qwen3.6 35B MLX	About 23–24 GB	Bulk summaries and text work; fast generation, but reasoning could consume a constrained output budget.
Qwen3.6 35B-A3B coding	About 23 GB in the agent test	Bounded code and mechanical edits; about 158 generation tokens/s in the measured configuration.
Qwen3.8 27B, eight-bit	About 30–31 GB loaded in the later tests	An alternative coding model and a stronger transcription draft in the small vision comparison; about 44 generation tokens/s in the coding probe.
Qwen2.5VL 32B	About 21 GB	Early vision drafts; later tests found problems with silent corrections, repetition and invented captions.
GLM-OCR	About 2.2 GB	A second reading to flag transcription disagreements; needed output limits and repetition handling.

Parameter counts describe the model, while the GB figures describe weights or loaded size in a particular configuration. Neither alone predicts task performance. The measurements came from this machine and these workloads; they are not promises for another deployment.

Give the local model a task it can finish

An early delegation approach asked a local model to return code as text. That was useful for sketches, but less effective when the code needed to land in a repository: the coordinating model still had to reproduce it in an edit.

The local coding agent changed that handoff. A scoped brief names the files, the intended behaviour and the checks. The agent works in a disposable Git worktree, edits the files itself and runs the specified tests. The coordinator then inspects the diff, verifies the result and applies the accepted changes.

The working sequence is straightforward:

1. A human and the coordinating model decide the task and acceptance criteria.
2. A local model implements the bounded change in an isolated worktree.
3. Tests and type checks provide mechanical evidence.
4. The coordinator reviews the changes and checks behaviour the tests do not cover.
5. Accepted changes are applied to the working project for the normal release process.

In one trial, we asked two models to clean up 49 comments across 22 files. The 35B coding model completed its run in 4 minutes 29 seconds; the 27B eight-bit model took 30 minutes 18 seconds. Both passed the tests and type checks. The faster model left a couple of awkward sentences. The slower model produced better wording but also edited two test titles outside the comments-only brief.

Passing tests established part of the result. Reviewing scope and meaning established the rest.

MCP made local capability available; CLAUDE.md made it part of the workflow

The integration uses a small Bun-based MCP server. MCP — Model Context Protocol — exposes tools that Claude can call during a task. In this setup, those tools send work to Ollama running on the same machine.

Tool	What it delegates
local_summarize	Reads a local file and returns a focused summary.
local_generate	Produces a text draft, or uses a vision model when images are supplied.
local_draft_code	Reads specified context files and returns a code sketch for review.
local_status	Checks the local server and available models.

For a large log or transcript, Claude can pass a file path and a question instead of first loading the entire file into its own context. The MCP server reads the file directly, sends its contents to the local model and returns the summary. Claude receives the tool call and result, and can inspect selected source passages when verification requires it. This avoids sending the whole source through Claude merely to decide which parts matter.

That mechanism does not decide which tasks to delegate. The workspace’s CLAUDE.md supplies the working policy: use local models for bulk summaries, repetitive transformations and well-specified work; keep architectural reasoning, security judgement and difficult debugging with the coordinating model. A corresponding AGENTS.md carries the policy for the other coding-agent workflow.

A condensed version of the policy looks like this:

1. For large logs and transcripts, pass a file path to local_summarize.
2. For repository edits with a decided contract, use local-agent.
3. State the files, required behaviour, checks and forbidden changes.
4. Inspect the diff and verify the result before applying it.
5. Keep architecture, security judgement and unresolved debugging with the coordinator.

The coding agent is a separate command-line tool, rather than one of those MCP tools. It handles the case where a text answer is the wrong deliverable: code needs to be written, checked and returned as a diff. Claude invokes the agent with a brief and test commands, then reviews its work.

Making tools available was not enough to get agents to use them. Delegation instructions needed to name the actual artifact and action: have the local agent write these fixtures, for example, then verify them. CLAUDE.md guides the model’s decisions; it is not an automatic scheduler or an enforcement boundary. The worktree, command restrictions and review process provide separate controls.

How much Claude capacity did this save?

The 49-comment trial illustrates where delegation can reduce Claude’s output.

Claude used roughly 3,000–4,000 output tokens to brief, check and review the local-agent run. Making the same edits directly was estimated to require 7,000–11,000 output tokens. That gives an indicative saving of 3,000–8,000 Claude output tokens for this task, using an estimated direct-edit baseline.

Earlier attempts to obtain complete code drafts through MCP exposed a different result: if Claude then had to reproduce the draft in file-edit commands, much of the output-token cost remained. Moving the actual edits into the local agent removed that duplication. MCP summaries address another source of usage by keeping the full input file out of Claude’s context unless it needs to inspect it.

Local-model token totals cannot simply be counted as Claude tokens saved. The two workflows may take different steps, use different tokenisers and perform different amounts of review. Setup, retries, tool results and corrections also consume Claude capacity. An API-price calculation would not directly describe savings on a subscription allowance.

To evaluate delegation in your own team, compare equivalent tasks with and without it. Track Claude input, output and cache usage, local processing, review time and the quality of the accepted result. That shows whether less model usage also translates into a better delivery process.

Smaller weights did not deliver the expected speedup

The next question was whether a four-bit version of the 27B model would make the slower workflow practical.

An initial comparison looked encouraging, but it had reused cached prompts. That inflated the apparent prompt-processing speed. The corrected experiment alternated models and used unique prompt openings to avoid that confounding factor.

Corrected comparison	Four-bit model	Eight-bit model
Generation with drafting enabled	48.5 tokens/s	44.0 tokens/s
Generation with drafting disabled	23.9 tokens/s	16.9 tokens/s
Processing a 9,606-token prompt	494 tokens/s	524 tokens/s
Processing a 45,580-token prompt	275 tokens/s	281 tokens/s

With drafting enabled, the generation advantage was only about 10%, and long-prompt processing did not improve. We kept the eight-bit version. This was a speed comparison; the four-bit model’s output quality was not tested.

The cache investigation also corrected an earlier interpretation. Reported input-token totals had suggested the agent was rereading everything. Inspection showed that 20 examined request transitions preserved the previous request as a prefix. A usage counter alone was not enough to establish how much work the inference server repeated.

Document reading needed its own acceptance criteria

The Otari project supplied a different workload: faithfully transcribing scanned technical manuals. Here, a plausible sentence could be a bad result. Printed errors, punctuation, numbers and structure needed to survive the conversion.

A six-page comparison used a hand-verified answer key with 534 words, 49 contents-list entries and five printed misprints. Qwen3.8, with thinking disabled, made 9 word errors out of 534 (1.7%). Qwen2.5VL made 47 (8.8%). The former preserved all five misprints; the latter preserved two.

GLM-OCR offered a useful independent reading but could repeat text until it exhausted its output allowance. Tesseract was quick and strong on the simple prose sample, but struggled with the dotted contents lists. No single reader won every part of the task.

A cross-check between Qwen3.8 and GLM flagged all six identified error spans in that small comparison while marking 9% of prose words and 4% of list words for inspection. That made it useful for directing attention. Agreement between two models still did not prove correctness.

The later full-manual work exposed the limit of that small test. Review found missing material, damaged heading and table structure, and incorrect figure crops despite passing automated checks. The workflow needed page-by-page comparison against the scans. The lesson was to test the complete deliverable, not assume a successful sample validated the whole pipeline.

Put review where it can change the outcome

This website’s photo lightbox is a recent example of the same delivery pattern. A local model produced the first implementation. Review then caught problems with the accessible dialog name, caption handling, scrolling and zoom behaviour. Browser checks—and feedback on the preview—shaped the final interaction before it was published.

That is the role of the stronger coordinating model: choose suitable tasks, investigate defects and judge the evidence. The local model contributes useful work within that process.

The toolchain restricts commands and separates work from the active checkout. Those controls help contain mistakes, but a test runner still executes code from the repository. It is not a hardened sandbox for hostile code. Similarly, local inference keeps delegated processing on the machine, but summaries or reviewed changes can still enter the cloud model’s context. A hybrid workflow needs an explicit data boundary.

What this demonstrates for enterprise delivery

The outcome is a working toolchain that assigns bulk processing and bounded edits to local models, while keeping architecture, debugging and review with the stronger coordinating model. It has been used for code changes, bulk text work and document drafts across the website and technical projects.

Evaluate the whole workflow: hardware, energy, setup, retries and review all have a cost. Measure completed tasks and the quality users receive, as well as token speed and model usage. Keep failed experiments too—they help explain when a different model or workflow is the better choice.

For an enterprise, the starting point is a workload with a clear boundary and a result someone can verify. The appropriate model, execution environment and review process follow from that workload.