On the evening of 22 August, Qwen3.5 9B was writing at about 55 tokens per second on BORZ, my gaming PC. A token is roughly a word or part of one. The model ran entirely on the graphics card, a Radeon RX 6700 XT with 12 GB of memory. Running AI models locally at speeds like that, on hardware I already had, was genuinely exciting.
I was curious what I could get out of local AI on that machine. I enjoy experimenting, understanding technology firsthand and finding out what’s practically possible, and BORZ had a decent GPU that wasn’t doing much.
AMD, Windows and Vulkan
I kept the machine as it was: Windows 11 Pro, no new hardware, no switch to Linux. For local AI that can be an awkward combination. Ollama, the runtime I used, didn’t list the RX 6700 XT among the cards it supported through ROCm, AMD’s compute platform.
It also supports Vulkan, and that’s what it used, without me changing a driver or a setting. A busy GPU didn’t seem like enough proof, so I checked the server log: library=Vulkan, and for the bigger model, offloaded 41/41 layers to GPU.
Finding the ceiling
Both models I started with were 4-bit quantized builds (Q4_K_M), which is what lets them fit. The 9B model used about 7.9 of the 12 GB. Qwen3 14B ran at about 35 tokens per second and used about 11 GB. When I doubled its context, the amount of text it can work with at once, from 4,096 to 8,192 tokens, the speed stayed the same and memory went up to about 11.6 GB.
So in what I tested, video memory was becoming the constraint well before speed was. Qwen3 14B at 8K context left very little of the 12 GB to spare. The next size up I looked at, a 27B model of about 18 GB, wouldn’t have fit on the GPU, and I left it alone.
From the Mac
By default Ollama only answers requests from the PC it runs on. One environment variable on BORZ opened it to the home network, and Ollama’s own startup shortcut picked that up after a reboot. I didn’t need a firewall rule or a router change. From the MacBook, over Wi-Fi, the 9B model ran at about 57 tokens per second, as fast as on BORZ itself. The only catch was that after a full reboot, someone has to log in to Windows before Ollama starts.
Talking to a model through curl and JSON isn’t something I wanted to do every day, so I installed Open WebUI Desktop on the Mac and pointed it at BORZ. On first launch it downloaded a 931 MB embedding model for features I hadn’t turned on. The first reply took about a minute to arrive. Later ones started within 4 or 5 seconds.
Then came the first real job: a simple Coming Soon page, one HTML file with the CSS and JavaScript inline. The 9B model wrote it in about 15 seconds, and it worked in the preview. Nothing about it needed a bigger model.
5,785 tokens for 2 + 3
Hugging Face has plenty of fine-tuned versions of these models, and Ollama can pull them directly. I tried two. The first, another Qwen3.5 9B variant, answered a few questions and then refused even a new chat asking what 2 + 3 is. The error said the request was 5,785 tokens, more than the 4,096 the model was loaded with. I removed it.
The second was OmniCoder 9B, a Qwen3.5 model tuned for software engineering. Same question in a new chat, same error, 5,775 tokens this time. When I asked it directly through Ollama on BORZ, it answered instantly.
That much I saw. The rest is inference. Two different models failed at almost the same request size, and the same question worked without Open WebUI. That points to the client adding several thousand tokens to each request before my question. I never established what those tokens were.
Raising the context to 8,192 got rid of the error, so I asked OmniCoder for a modern Coming Soon page with a calendar block and an email signup. The first attempt never produced a final answer. The second looked substantial, but the browser showed only a background gradient. The file stopped inside a CSS media query, before any page content. If the request was still around 5,800 tokens, that left about 2,400 for the page, which would explain it.
At 16,384 the model still fit entirely on the GPU and started answering in about 2 seconds. This time it didn’t run out of room. It wrote calendar days instead: 20, 21, 22, up to 31, then 1, 2, and around again, for minutes, until I stopped it. It never reached </html>.
I could have tuned its sampling settings until it behaved. I didn’t want to rescue a model that hadn’t yet shown me why it was worth rescuing.
Just after midnight
That’s when I stopped. The plan had been to keep the setup within reach and see whether I’d still choose it for real work once the novelty wore off. I didn’t need to get that far. For practically everything I’ve wanted to do, Claude and Codex have been substantially better. That’s my experience, not a controlled comparison.
Speed wasn’t what held it back. 35 to 57 tokens per second is comfortable to work with. What held it back was what fits in 12 GB, and the friction around it: a slow first reply, context settings, and a chat app whose requests were far larger than my questions. I didn’t measure what it cost to run, so I can’t say anything useful about that.
Ten days later I gave the local models one more job, where getting it right mattered more than speed: classifying short, structured work notes against a small rubric. Across six reviewed cases, Qwen3 14B got two exactly right. Qwen3.5 9B got one, and its run took about 100 seconds per case. A hosted agent did much better on the same cases. That wasn’t a clean comparison of models, though. The hosted agent could read the full records, far more context than I’d given the local ones.
What I kept
I came away understanding the stack much better: how quantization decides whether a model fits, how context eats into memory, and how to tell a failing model from a failing client.
The agents I run on BORZ for remote development still send their requests to cloud models. Local AI didn’t offer enough for my work yet, but small, disposable jobs like that first page are where it did fine.
On my setup, an RX 6700 XT on Windows with Ollama and Vulkan, Qwen3.5 9B at Q4_K_M ran comfortably and Qwen3 14B at Q4_K_M fit with up to 8K context. And if a model misbehaves in a chat app, try it through Ollama directly before blaming the model.