Tag: grok

Let’s try the latest open weights models

A friend of mine recently asked me to test my current autonomous LLM game generation pipeline with a few of the latest open weight models. I used Visual Studio Code GitHub Copilot and it’s “Bring Your Own Key” capability to hook it up to the various models on my Open Router account.

I was a bit underwhelmed by the models I tested (DeepSeek 4 Pro, GLM 5.2, and Kimi K 2.7-code), at least for my game generation prompt. It should be noted that, just because one run on my workload resulted in underwhelming results, that doesn’t mean that these models wouldn’t do well on someone else’s coding workload, or even on another run.

Interestingly, just as I was wrapping up my testing, Moonshot AI released Kimi K-3. The drop of K-3 instantly generated a lot of hype, and it was being compared to Opus and Fable on various social media (especially YouTube). I wanted to test it out on my own workload.

Easier said than done, as when I submitted my prompt to Kimi K3 through OpenRouter, the prompt would “bounce”, with a message that seemed to be indicating that K3 was being throttled due to excessive demand, and indicating that I should “try again”. After trying at various hours throughout the day and evening, I resigned myself to waiting until the initial demand had dissipated before trying agin.

Rate limit message from Kimi-K3

Luckily the next day I was able to run the prompt, and lo and behold Kimi K3 generated a nice, playable game autonomously. The game it generated is comparable to ones generated by Fable, Opus 4.8, Sonnet 5, Grok 4.5, and GPT-5.6 Terra and Sol. It didn’t seem to be particular speedy, but cost-wise it produced the finished game for ~$13, as opposed to ~$50-$80 for the Anthropic models.

From my tests, I would rank Kimi K-3 below Fable, Opus 4.8, and Grok 4.5, but above Sonnet 5 and GPT-5.6 Sol and Terra, which is quite amazing. In fact, the rise in ability of Grok 4.5 and Kimi K-3 in these types of tests is truly amazing. A reminder, the prompt has the agent implement AND design the features, so I am not only testing coding, but the “aesthetic” ability of the models.

Run-able versions of the open weight models’ generated games on a new website.

The run-able versions of the games generated by the open weight models can be found here:

https://rgm-llm-asteroids-openweight-arcade.netlify.app

You can compare the results of the open weight models to the games previously generated by the frontier LLMs:

https://rgm-llm-asteroids-arcade.netlify.app

I still like the design chops of Fable, but I can’t justify its cost for a project like this. I’ve run some tests with using Fable for explicit game design and project scaffolding, and then switching to Grok 4.5, GPT-5.6, or Sonnet 5 for the code implementation and testing phases, all in the same context. The initial results are encouraging, and I’ll post another blog post soon with the details.

The models get muscle

Things have been moving fast. SpaceXAI dropped Grok 4.5 and OpenAI released GPT-5.6 Sol, Terra, and Luna. For me the real surprise was Grok. I’d been using it mostly for research and had never reached for it on coding work — Claude Sonnet had been my default for that. My quick-and-dirty way of checking whether a model was actually useful was a one-shot prompt that asked it to build a browser-based clone of the 1979 vector Asteroids game.

I picked that old game for a few reasons that still hold up. It touches on a lot of what makes games interesting: graphics, basic physics, sound, scoring, projectiles, collisions, and even a little destruction effect when the ship blows up. At the same time it’s simple enough to describe without locking the model into every last detail, so you can see what it decides on its own for the ship, rocks, and UFO. JavaScript is one of the languages these models have seen the most of, and the whole thing runs in any browser with zero install.

Back in March 2026 I ran Grok 4.1 through that same prompt and it was pretty rough. Asteroids came out warped, the ship thrust sideways instead of forward, you’d hear the shot sound but nothing actually appeared on screen. It was one of the weaker results I got at the time.

Grok 4.2 Auto
GPT-5.3 Codex

Around then I started putting together a multi-agent setup — an orchestrator that would hand work to a planner, then a coder, then an auditor, and so on — so the models could build a small game feature by feature.

Before I had the whole thing tuned the way I wanted, the newer models (Grok 4.6 and GPT-5.5 among them) got noticeably better at just working through things on their own. They could plan, implement, test, and even come up with the next feature without constant hand-holding. I ended up shelving the orchestrated pipeline for a while.

The other big jump has been in vision. These models are now decent at looking at a screenshot of the game and checking whether what they think is happening actually matches what’s on screen. That self-verification step removes a lot of the back-and-forth that used to require a human in the loop.

To keep testing meaningful I put together a tougher prompt. On top of just building Asteroids, the new version asks the model to:

  1. Follow SOLID and KISS/YAGNI principles where they make sense.
  2. Use test-driven development for the pieces where it’s practical.
  3. Avoid the usual LLM shortcuts when it comes to testing.
  4. Wire up Playwright so it can actually verify graphics, physics, and on-screen behavior.
  5. Support custom game modes (single-step mode has been especially useful for debugging).
  6. Come up with 25 new features that feel fun and replayable, then plan, build, test, and commit each one to git in sequence.

Some of the games available on the game examples web page, generated by LLMs.

I’ve put the playable game results from the latest runs here, in a game examples web page:

https://rgm-llm-asteroids-arcade.netlify.app.

‘c’ and ‘1’ to insert coin and start the game.

‘left arrow’ and ‘right arrow’ to turn

‘forward arrow’ to thrust

‘space’ to fire

‘h’ to hyperspace

Example of a game designed, generated, and tested by an LLM

Every game is a full browser version of Asteroids with 25 new features the model designed itself. I’ve also included two earlier builds (the ones with the gold background) that only had 20 features. I liked the feel of those two enough that I wanted to show what was possible with this single prompt pipeline.

Lastly, I’ve included the prompt used to make these games at the bottom of the game examples web page. Feel free to use it as inspiration for your own prompts.

One practical note: running these long agentic loops eats tokens, especially once the context grows. Grok 4.5 and GPT-5.6 Terra Medium were both quick — around 15-20 minutes per game — and Grok barely touched my monthly token budget. Claude Sonnet 5 took an hour and forty minutes and ran up $112 in tokens. Opus 4.8 and Fable sat in the middle at about 45 minutes and roughly $60 each.

The frontier models have reached the point where they can keep iterating on a messy, ambiguous problem until it’s done. The older and smaller models still tend to stop after the first pass and wait for you to say “go do the next feature.” I’m planning to run a side-by-side on the single-agent approach versus the older orchestrator setup to see how they compare on speed, cost, and context size.

Giving the model a way to look at its own output visually has been one of the bigger practical wins. The more ways you can let it check its own work, the fewer obvious mistakes make it into the final build. The long-term interesting question is whether these models can eventually judge whether a feature or a whole game actually fun. That’s going to take more experimentation.