Local vs Frontier AI
DNSai.com
Author: AI Security
Share

Local vs Frontier AI: 8 Local Models vs Claude, GPT and Grok

Every few months the question "how good is AI now?" gets a new answer. Rather than read another leaderboard, I wanted to see it for myself, with the same brief for every model and the results on screen side by side.

So I ran a small experiment. Eight open-weight models ran locally on a single PC in my office, with no cloud and no internet connection. Three frontier models ran on their vendors' own infrastructure. All eleven got the same two programming tests, and I recorded what each one built.

The local machine

The local machine is not exotic. It has an AMD Ryzen 9 5900X (12 cores, launched November 2020), 128 GB of RAM, and an NVIDIA RTX 3090, a 24 GB graphics card that NVIDIA announced in September 2020 at $1,499. It runs Ubuntu, and the models run on free, open tools: Ollama, llama.cpp, and Strata, a runtime that keeps a large model's "experts" in system memory. One model runs at a time.

Hardware card: RTX 3090 24 GB, Ryzen 9 5900X, 128 GB RAM, Ubuntu, fully offline
The whole local setup: one 2020-era PC, no network connection needed while the models work.

Two of the local models are far too big for the graphics card. DeepSeek V4-Flash has 284 billion parameters and Qwen3.8-Flash-Next has 125 billion. They run anyway, with most of their weights in system RAM, at 13 and 54 tokens per second. That's slower than a cloud service, but nobody else ever sees the prompts.

Here's the two-minute version of everything that follows. Every clip is labelled with the model that wrote it, and tagged LOCAL or FRONTIER. The calm ambient soundtrack was generated locally on the same machine.

Local vs frontier: eleven models, two tests, the same brief. Every frame was rendered from pages the models wrote. Music generated locally with ACE-Step. Watch on YouTube.

The contestants

Table of the eleven models: eight local models with maker, country and size, and three frontier cloud models
The eleven models, who makes them, how big they are and where they ran.

The local side. Five of the eight local entries come from Chinese labs: four builds of Alibaba's Qwen, and DeepSeek V4-Flash from DeepSeek, a Hangzhou company owned and funded by the hedge fund High-Flyer. That's simply what the open-weight world looks like now. Hugging Face reported in March that Chinese models accounted for 41% of downloads on its platform over the past year. The American local entries are Google DeepMind's Gemma 4 31B and the fast Gemma 4 26B-A4B, the first Gemmas released under Apache 2.0, and Meta's Muse Glimmer 30B, an open model distilled from Meta's closed Muse Spark. A ninth local model, Microsoft's 3.8-billion-parameter Phi-4-mini, produced nothing that worked and is left out of the comparison.

The frontier side. Anthropic's Claude Opus 5.5 and OpenAI's GPT-6 Astra ran through their vendors' official command-line tools, with web search and every other tool switched off, so they worked from the brief alone, just like the local models. xAI's Grok 4.7 was harder to pin down; more on that below.

Test 1: a Swiss cheese, cut with real geometry

The first test was deliberately precise. Each model had to build an interactive 3D block of Swiss cheese in a single web page, with the holes actually cut out of the geometry using a 3D boolean library rather than painted on. Each model got the prompt three times, with up to three chances per try to fix errors the browser reported.

Grid of Swiss cheese renders from all eleven models, labelled local or frontier
The best cheese from each model. Frontier models are outlined in blue.
  • Reliability. On the original prompt, the seven local models I compared (Flash-Next joined later, and Phi-4-mini is left out) produced a working page on their first answer in 12 of 21 tries, and in 20 of 21 after fixes. Claude Opus 5.5 and GPT-6 Astra worked first time in 5 of 6 tries, and in all 6 after one fix.
  • Speed. Claude and GPT took about 2.5 to 5.5 minutes per try. The best local model, the 27-billion-parameter Qwen3.8 run through llama.cpp, took 18 to 75 minutes because it thinks for a long time. But its best cheese, a block of Emmentaler with eighteen carved holes and a museum-style title card, earned the only perfect score from a blind judge.
  • Code quality. I ran an automated code review on every page. The frontier pages, Grok's included, had no errors. In 19 of 24 local pages, the code made the same mistake: it gave the hole-cutting shapes no material, which leaves the inside of every hole a flat, unlit white.
Best local cheese (Qwen3.8 27B) next to Claude Opus 5.5, GPT-6 Astra and Grok 4.7
The best local cheese (top left) against the three frontier models.

That last finding is the useful one. The gap between local and frontier models isn't mysterious. It's a list of specific, learnable habits. I've turned what the frontier code does differently into a written coding standard for the local models to follow next.

The Grok problem

Grok 4.7 was the most frustrating model in the test. Through xAI's command-line tool, it ignored the instruction not to use tools and fetched the geometry library's source code from the internet. In other attempts, it announced what it was about to do and then stopped without writing anything. In the chat app, it kept saying it was about to finish the cheese without delivering the file, and it took repeated nudging to get one working page. That page is in the grid, but it's a single manual entry, not three automated ones.

Test 2: the Huey

The second test had no right answer. The brief: build a cinematic, animated 3D showcase of a Bell UH-1 Iroquois, the "Huey", flying over Vietnam during the war, and the more detail the better. There was no time limit and no token limit. Each model got three passes to build, finish and polish its scene, and each final scene was recorded as a 30-second 1080p video.

Grid of Huey helicopter scenes from all eleven models, labelled local or frontier
A frame from each model's Huey scene. Frontier models are outlined in blue.

Among the local models, Qwen3.8-Flash-Next was the clear winner. It built open cabin doors with a crew chief and a door gunner, the rotor's stabilizer bar, skids, U.S. Army markings, haze over karst mountains, and a five-shot camera sequence. The 27B Qwen3.8 and DeepSeek V4-Flash came next. All of the top local entries were Chinese and open-weight.

Best local Huey (Qwen3.8-Flash-Next) next to Claude Opus 5.5, GPT-6 Astra and Grok 4.7
The best local Huey (top left) against the three frontier models.

The frontier scenes are a visible step up: proper lighting and shadows, believable terrain, rice paddies, rivers and misty karst hills. Claude Opus 5.5 and GPT-6 Astra completed all three passes in under an hour each, and every pass rendered. Grok 4.7 ran once through its command-line tool with a tighter brief, and thought for 38 minutes before writing a single, detailed file.

The twist: the best-looking local helicopter flew backwards

When I watched the local recordings again, something was off. Some of the helicopters were flying backwards. So I stopped eyeballing and measured: a script runs each page, finds the helicopter, and compares where its nose points with where it is actually going.

Flash-Next's helicopter, the best-looking local one, flew tail-first 100% of the time. So did Muse Glimmer's. Two rounds of blind AI judging had scored these videos without catching it.

I gave each model its own measurement and one paragraph about the likely cause, a mix-up over which way the coordinate axes point. Flash-Next and Muse fixed it in a single pass: 100% nose-first, banking correctly into their turns.

Before and after: Qwen3.8-Flash-Next's helicopter flying tail-first, then nose-first
Before: tail-first. After one measured fix: nose-first.
Chart: share of flight time each local helicopter flew nose-first, before and after the fix pass
Share of flight time each local helicopter's nose pointed within 30° of its direction of travel, before and after the fix pass.

Grok's brief included the flight rules the local models had to learn the hard way. Measured with the same script, its helicopter flew nose-first, upright and banked into its turns 100% of the time, with the main rotor turning the right way at the right speed. It was the only entry to get every flight measurement right in one go. It was also by far the heaviest page: about 7 frames per second on the RTX 3090.

That's the lesson I'd hand to any security team. "Looks right" is not the same as "is right", and the only way to know the difference is to measure.

The gap is measured in months

The frontier models won this test. But how far behind is local AI really? Epoch AI estimates that the best open models runnable on a single consumer GPU match frontier capability after 6 to 12 months. For the best open models of any size, Epoch put the gap at about four months in May.

These are the labs' own numbers, measured with different setups, so treat them as a direction rather than a photo finish. The direction isn't in doubt. What runs locally on a 2020 graphics card today beats what the best paid AI could do two years ago. The frontier keeps moving too, so the whole curve rises, and it rises for everyone.

Everyone gets the upgrade

That includes the people you'd rather didn't. The same improvements that make AI useful to your team make it useful to attackers, whether they rent a frontier model or download an open one and run it where nobody can see.

  • In May, Google's Threat Intelligence Group reported the first zero-day exploit it believes was developed with AI.
  • Earlier in the year, Google described attackers using language models for "hyper-personalized, culturally nuanced lures."
  • Unit 42's June "Phantom Squatting" research asked AI models about 913 brands. The models invented web addresses that don't exist, and roughly 250,000 of those domains were still unregistered. In one case, 23 days after Unit 42 flagged a predicted look-alike domain for a national postal service's online marketplace, an attacker registered it and put a card-stealing phishing kit on it.

Watch your domains and the look-alikes you know about

DNSai Domain Manager is built on a living record of millions of domains, and it reads the public DNS of the domains you add. Each one gets a full profile: mail provider, SPF, DKIM and DMARC posture, hosting, the SaaS platforms behind it, and every change over time, with alerts when something changes. Add every domain you own, plus subsidiaries and the look-alikes you already know about, and it starts scanning and profiling within minutes. It watches the list you give it. It does not go looking for newly registered look-alike domains on its own.

Explore Domain Manager →

You can't control which AI an attacker uses, or how good it will be next quarter. You can keep watch on what it would be aimed at: changes to your own DNS, changes on the look-alike domains you already track, and gaps in email authentication that a phishing kit would love. AI is improving for everyone, so everyone needs to level up the same two things: visibility into what's happening around their domains, and the intelligence to act on it. And, as the helicopters showed, the habit of measuring instead of assuming.

How I tested

Every local model ran alone on the same machine with no output caps. The frontier models ran through their vendors' official tools with web search and other tools switched off. Grok 4.7's cheese was entered by hand through its chat app, and its helicopter came from a single pass with a brief that already included the flight rules, so treat its results as indicative. Local judging was blind: an AI judge scored anonymised screenshots and videos. Flight direction was measured by a script, not judged. Each test was a small number of runs per model, so small differences are noise. The local models ran as compressed ("quantised") builds, which score a little below the full-precision weights the labs benchmark. The generated code is not being published.

Sources

  1. NVIDIA Delivers Greatest-Ever Generational Leap with GeForce RTX 30 Series GPUs — NVIDIA Newsroom, September 1, 2020. nvidianews.nvidia.com
  2. AMD reveals Ryzen 9 5900X CPU — Shacknews, October 8, 2020. shacknews.com
  3. State of Open Source on Hugging Face: Spring 2026 — Hugging Face, March 17, 2026. huggingface.co
  4. DeepSeek — Wikipedia, accessed October 28, 2026. en.wikipedia.org
  5. Qwen3.8-27B model card — Qwen (Alibaba), 2026. huggingface.co
  6. Qwen3.8-Flash-Next model card — Qwen (Alibaba), 2026. huggingface.co
  7. Qwen3.6-35B-A3B model card — Qwen (Alibaba), 2026. huggingface.co
  8. DeepSeek-V4-Flash model card — DeepSeek, 2026. huggingface.co
  9. Gemma 4: Byte for byte, the most capable open models — Google, April 2, 2026. blog.google
  10. Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device — Meta, August 10, 2026. research.meta.ai
  11. Phi-4-mini-instruct model card — Microsoft, February 2025. huggingface.co
  12. Frontier AI performance becomes accessible on consumer hardware within a year — Epoch AI, August 15, 2025. epoch.ai
  13. Open models lag state-of-the-art closed models by 4 months — Epoch AI, May 29, 2026. epoch.ai
  14. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku — Anthropic, October 22, 2024. anthropic.com
  15. Introducing Claude Sonnet 4.5 — Anthropic, September 29, 2025. anthropic.com
  16. simple-evals (GPQA Diamond results for o1-preview and GPT-4o) — OpenAI, GitHub, accessed October 28, 2026. github.com
  17. GTIG AI Threat Tracker: Distillation, Experimentation, and (Continued) Integration of AI for Adversarial Use — Google Threat Intelligence Group, February 12, 2026. cloud.google.com
  18. GTIG AI Threat Tracker: Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access — Google Threat Intelligence Group, May 11, 2026. cloud.google.com
  19. Phantom Squatting: AI-Hallucinated Domains as a Software Supply Chain Vector — Palo Alto Networks Unit 42, June 30, 2026. unit42.paloaltonetworks.com

Share this article

Related Articles