An OpenAI-compatible endpoint on localhost:8080 is all it takes to turn an agentic coding tool into a fully offline one — here is what the stack actually does, and where it leaks.
Background
Agentic coding assistants are, architecturally, thin clients. They manage a conversation, a tool loop, and a file system diff — and they hand the hard part to a remote /v1/chat/completions endpoint. That endpoint is the only thing they truly depend on. Which raises an obvious question: if the protocol is just OpenAI’s REST shape, why does the model have to live in someone else’s data center?
On Apple Silicon it doesn’t. mlx-lm ships a server that speaks the same wire format, backed by MLX, Apple’s array framework for the unified-memory architecture. Unified memory is the whole point here: on a discrete-GPU machine, weights must be copied across PCIe into VRAM, and a 32B model at 4-bit (~18 GB) simply won’t fit on most consumer cards. On an M-series chip, the GPU addresses the same physical DRAM as the CPU, so a 4-bit quantized 32B model is a memory-capacity problem, not a VRAM problem. That’s the constraint that flipped.
I built asciifun to test the loop end to end: local model, local agent, and a small, verifiable task — image-to-ASCII conversion in two languages.
How It Works
Three moving parts, and one of them is doing something less obvious than it looks.
The server. mlx_lm.server --model mlx-community/Qwen2.5-Coder-32B-Instruct-4bit pulls the quantized weights from Hugging Face and exposes /v1/models and /v1/chat/completions on port 8080. Quantization here is group-wise affine (4-bit weights, fp16 scales/zero-points per group), so the model dequantizes on the fly in the matmul kernel. Prompt processing is compute-bound; token generation is memory-bandwidth-bound — which is why an M-series machine with high memory bandwidth generates tokens at a rate that’s usable, while prefill on a long context is where you’ll actually feel the wait.
The client. OpenCode has no MLX integration and doesn’t need one. It has a provider abstraction backed by @ai-sdk/openai-compatible, so you declare a provider whose baseURL points at the local server and it treats it exactly like any hosted API. No shim, no proxy.
The generated code — the interesting part. I asked the local model for an ASCII converter in Swift. What it produced doesn’t iterate over source pixels at all. It creates a CGContext at the target character-grid resolution and calls context.draw(cgImage, in:), letting CoreGraphics rasterize the full-resolution image down into that tiny bitmap. The downsampling filter — the averaging of thousands of source pixels into one cell — is done by the rasterizer, in optimized code, for free. The script then walks a width × height × 4 buffer where every pixel is already one ASCII character. The Python variant does the same via Pillow’s Image.resize. Same idea, different rasterizer.
Code
let chars = "@%#*+=-:. " // dark → light: 10 luminance buckets
let aspect = CGFloat(cgImage.height) / CGFloat(cgImage.width)
// 0.55 compensates for terminal cell geometry: a monospace cell is
// roughly twice as tall as it is wide, so we squash the row count
// or the output comes out vertically stretched.
let height = Int(CGFloat(width) * aspect * 0.55)
// Draw the image into a bitmap that IS the character grid.
// CoreGraphics does the box-filter downsampling.
let context = CGContext(
data: nil,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: width * 4,
space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue
)!
context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height))
let pixels = context.data!.bindMemory(to: UInt8.self, capacity: width * height * 4)
for y in 0..<height {
var line = ""
for x in 0..<width {
let i = (y * width + x) * 4
// BT.601 luma coefficients — perceptual weighting, not a naive mean
let gray = UInt8(Float(pixels[i]) * 0.299
+ Float(pixels[i + 1]) * 0.587
+ Float(pixels[i + 2]) * 0.114)
let idx = min(Int(gray) * chars.count / 256, chars.count - 1)
line.append(chars[chars.index(chars.startIndex, offsetBy: idx)])
}
print(line)
}And the entire OpenCode wiring:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"mlx": {
"npm": "@ai-sdk/openai-compatible",
"name": "MLX (local)",
"options": { "baseURL": "http://127.0.0.1:8080/v1" },
"models": {
"qwen2.5-coder-7b": {
"name": "Qwen2.5-Coder 7B (local MLX)",
"limit": { "context": 32768, "output": 8192 }
}
}
}
}
}Run it: swift ascii.swift photo.jpg 60. No package manifest, no Xcode project — swift in interpreter mode compiles and runs the file directly.
Trade-offs & Limitations
The character ramp goes dense-to-sparse, mapping low luminance to @. That’s correct on a light background and inverted on the dark terminal most of us actually use — the output reads as a photographic negative until you reverse the string. CGContext is also hardcoded to deviceRGB with premultiplied alpha, so a PNG with transparency composites against black, not white. And the model id in opencode.json and the one the server was launched with need to agree; mismatched names are the fastest way to a confusing 404. On quality: a 4-bit 32B coder model is competent at self-contained scripts like this one and noticeably weaker at multi-file refactors, where the context window and the quantization loss compound.
My Take
The generated code is better than I expected in one specific way — offloading the downsampling to the rasterizer is the move an experienced developer makes, and it’s the move a naive nested-loop implementation misses. But the value of this setup isn’t the ASCII art. It’s the demonstration that the agentic part of an agentic coding tool is client-side and portable, and the model behind it is a swappable dependency reachable over 127.0.0.1. For code that touches a client’s proprietary source — the kind I write most weeks — that distinction is not academic. Run this on anything with enough unified memory and the confidentiality question stops being a question. Watch the prefill latency, not the tokens-per-second: on a long file, that’s where local inference stops feeling free.
Tags: Swift, MLX, Apple Silicon, Local LLM
