Jev vs. Claude vs. GPT Mini: Who Lasts Longer in the Dino Game?
If you hook an AI up to the dinosaur game, can it look at obstacles and pick the next move in time? We split general-purpose models and decision-only APIs into two rounds and recorded real gameplay. In the first round, Jev survived to 60 seconds; in the second, Jev and OpenAI Decisions both survived to 5 minutes. But is the model that answers faster also the one that decides more accurately? After walking through how we connected everything and how the costs compared, we look at what happened when we waited for the answers of seven models, including Solar, and measured accuracy and API response time separately.
What we wanted to test with the dino game was whether decisions arrive in time
In a game, you have to pick the next move while an obstacle is approaching, so response latency is something you can actually see. At first, having the AI look at the screen and pick everything down to the jump timing was hard. So we started from the public Jev dino game code and split state reporting from execution.
In this experiment, the AI doesn't get a screenshot. The game code reads the obstacle type, count and height and the current speed and passes them on, and the model picks one of jump, duck or keep running, plus a jump type. The code takes that choice and calculates when to actually press the keys.
For example, when an obstacle approaches, the AI chooses whether to jump. When to press the key and how long to hold it is up to the shared control code. Since the AI isn't pressing keys every frame, how well the game goes depends on both the model and the control code.
We gave three models the same state and options
We specified Jev 1.13, Claude Haiku 4.5 and GPT-5.4 Mini. Claude and GPT returned one of the fixed options via function calling through OpenRouter's Chat Completions, and Jev used the choice questions in the Decisions API. We used the same state, question and options in meaning, but converted the API format to fit each model.
| Model as shown on screen | Model ID used in requests | Response method |
|---|---|---|
| Jev 1.13 | typesafe/jev-1.13 | Fixed choice via the Decisions API |
| Claude Haiku 4.5 | anthropic/claude-haiku-4.5 | Function calling via Chat Completions |
| GPT-5.4 Mini | openai/gpt-5.4-mini | Function calling via Chat Completions |
We built the connection in this order: reading game state, a per-model API adapter, then shared control code. First we checked that state and actions made the round trip with one model, then changed the others to return the same options. API keys were read on the server, not put in the browser.
Below are the core requirements to use when having an AI coding tool write the connection code. For exact API endpoints and response formats, check the TypeSafe docs, the OpenRouter docs and the official GPT-5.4 Mini docs.
Read the obstacle state from the public dino game code and send it to the API.
The model chooses jump / duck / keep_running and a short / full jump.
Have the shared code calculate the jump timing from the current speed.
Even if the dino is mid-jump at the first observation, don't assume it stays that way until the obstacle arrives.
Only allow a short jump for a single small cactus.
Record and replay the same obstacle types, counts, heights, gaps and speed adjustments.
Don't pause the game while waiting for the API; run it at the original speed settings.
Log API errors separately from crashes, and don't use fallback actions or mock responses.
End the test at 60 seconds of real time and log response time, cost, model name and failure reasons.
We turned reasoning off for Claude and GPT and set temperature 0 with a 128-token output limit. We didn't use automatic retries, response caching or fallback actions that hide API failures. We also didn't change whether to act based on Jev's confidence number.
The same seed wasn't enough, so we fixed the obstacle course itself
At first we thought the same random seed would make for a fair comparison. But the order in which random numbers get consumed at runtime sometimes changed obstacle gaps or types, so we recorded the properties of the course that was actually generated and replayed it for the other models. All three models in this recording met the same first obstacle, 3 small cacti, and we confirmed that the generated obstacle records matched the reference course.
Game settings were base speed 6, acceleration 0.001 and max speed 13. Runs took place in real time in the browser, and the game didn't pause while waiting for the API. The original game's acceleration depends on frame count, so the results can't be generalized to other computers or browsers.
For the Reel, we picked each model's longest-surviving recording
In the first edit, Haiku ended at the first obstacle, so its part of the comparison was too short. Past experiments had a run in the 14-second range, but no video of it survived, so we recorded additional real API play on the same course with the same questions and control code. We didn't change the game speed or obstacle rules, or stretch the footage to make anything look like it lasted longer.
The valid recordings were 7 runs for Haiku, 7 for GPT Mini and 4 for Jev. Haiku and GPT each include 4 runs on default routing and 3 on OpenRouter's latency-first routing. Routing is the setting that decides which provider's server an API request goes to. All 4 Jev runs survived until the test ended at 60 seconds of real time. The total cost recorded in the API responses across all 18 valid runs, including the first recording, was $0.047368706.
| Model | Runs recorded | Best result | Avg. API response in selected run | Calls / cost in selected run |
|---|---|---|---|---|
| Claude Haiku 4.5 | 7 | 15.38 s / crash | 1135ms | 10 / $0.01368800 |
| GPT-5.4 Mini | 7 | 17.19 s / crash | 994ms | 13 / $0.00540075 |
| Jev 1.13 | 4 | 59.13 s game time / survived to 60 s, test ended | 395ms | 59 / $0.00169180 |
The selected Haiku run used provider.sort: latency, and the GPT Mini run used default routing. According to OpenRouter's official docs, this setting asks to prioritize low-latency providers; it doesn't guarantee response time across a whole game. Jev used its dedicated Decisions API.
The Reel is a highlight showing the best recorded play. Each model had a different number of attempts and the routing was mixed, so don't read it as a fair win-rate or average-latency comparison or a universal performance ranking. Call costs also differ with play length and call count, so we don't calculate savings on the same task from the totals alone.
Round 2 ran decision-only APIs against each other for up to 5 minutes
After the first comparison, we compared APIs that return a choice instead of general-purpose models. We connected Jev, OpenAI Decisions (GPT-6 Luna Decisions), MS Decision-1 and Clef Flash to the same obstacle course and control code, and ran each model 5 times for up to 5 minutes of real time. MS was recorded separately under the same conditions after the other three. Solar, originally included, was left out of the comparison in this Reel.
The Reel shows each model's longest-surviving run. Jev and OpenAI Decisions both made it to the 5-minute cap without crashing. That means both models survived to the end in their best run of this test. Across all 5 runs, Jev reached the cap twice and OpenAI Decisions once; MS and Clef never did. This doesn't establish accuracy on business tasks or superiority across all games.
Solar wasn't so much wrong as late
At first we also connected Solar Decide to the same game. All 5 valid runs crashed into the first obstacle at about 4.48 to 4.49 seconds of game time. But when we checked the API responses, it had chosen jump, full jump all five times. That was the right choice for clearing 3 small cacti, but the responses took 5.19 to 8.85 seconds, with a median of 6.07 seconds. The game doesn't pause while waiting for the API, so by the time the answer arrived, the crash had already happened.
So describing this result as "Solar misjudged the obstacle" isn't accurate. It returned the right choice for the first obstacle it observed, but the answer didn't arrive in time, which made it hard to use for this real-time game connection. Since it never got past the first obstacle, this also isn't an evaluation of its judgment on the various obstacle types that come later.
Would it be faster called directly instead of through OpenRouter?
On October 15, 2026, we sent the same request alternately to the Upstage direct API and to Solar Decide on OpenRouter. We used the game's first-obstacle state and the two questions as is, and both sides had 1,031 input tokens. We ran one request at a time on the same computer and measured 10 times per route, excluding 1 warm-up each. Timeout was 15 seconds, with no automatic retries.
| Result | Upstage direct API | OpenRouter |
|---|---|---|
| Requests measured | 10 | 10 |
| Successful responses | 9 | 10 |
| Median time for successful responses | 5.30 s | 6.84 s |
| Range for successful responses | 3.41 to 11.98 s | 3.40 to 14.48 s |
| 15-second timeouts | 1 | 0 |
| Choice in successful responses | All jump, full jump | All jump, full jump |
The median and range are calculated over successful responses only, and timeouts are shown separately as failures. The direct API's median for successful responses was about 1.54 seconds shorter, but delays of several seconds and a timeout showed up with direct calls too. We can't attribute the whole difference to OpenRouter's relay time. Server load can vary, and the direct API only returned the alias solar-decide, so we couldn't confirm whether its internal model version matches OpenRouter's upstage/solar-decide-20260928.
Download per-route call measurements (CSV)
So is it a network problem rather than the model?
What we measured is the total time from starting to send the request until the full response body is received. That includes not just network communication but also time spent waiting in a queue on the server, input processing and the time the model takes to compute an answer. Observability tools for inference servers also track queue time and model compute time as separate metrics (NVIDIA Triton docs). That doesn't mean Upstage uses that server.
We didn't get per-stage timings from inside Upstage, so we can't separate whether the problem is the network, server queuing or slow model compute itself. We confirmed the delay reproduced with direct calls, but that alone doesn't mean we measured the model's pure inference speed. Decision accuracy and end-to-end service response time need to be read separately.
Could it have done well at the game if run locally?
It's possible. If the same model ran on dedicated hardware fast enough that the answer arrived before the jump point, it might clear the first obstacle we observed. Running locally can cut external network round trips and shared-server queuing, but model compute can also be slower on a personal computer, so it isn't automatically faster. An API is just a way of calling a model, so a local model can be served through an API too. What matters is less the name of the call and more where the model runs, on what hardware and under what conditions.
We didn't run Solar Decide locally this time. The official model page lists Upstage Console as the way it's offered, so you'd first need to check separately whether the same model's weights and decision implementation are available for local use. Results from installing some other open model locally can't stand in for Solar's local performance either.
The conclusion we can support here is "Solar's first decision was right, but with this API connection, the answer didn't arrive within the time the game needed." Whether it would have lasted reliably if run locally can only be known by running it again with the same model and control code.
Given time to answer, Solar also got all 18 states right
In a real-time game, a correct answer that arrives late is useless. So what if we pause the game and wait for the answer? On October 15, 2026, instead of playing the game in real time, we added an evaluation that gives each model the same game state and waits for its answer. In this evaluation, a slow response doesn't cause a crash or count as wrong.
The inputs were the obstacle type, count and height category prepared by the code, the speed, and what the dino was doing at the moment of observation. The correct answer for each state was fixed in advance using the existing action rules: jump for ground obstacles, duck for birds that only block a standing dino, keep running for birds that won't hit even a standing dino. The question also spelled out the selection criteria, so this is closer to a task of picking an action by given rules than to complex reasoning or screen recognition.
We originally planned 24 states, 8 each for jump, duck and keep running. But Claude's call on the 19th state got an HTTP 402 for insufficient credits and we stopped, so we compared only the first 18 states where all seven models responded normally. That shared set has 6 states per action. The 19th state, which only Solar answered, and the 402 response are left out of the table, and we didn't mix in results from a separate run that was stopped earlier by a shared connection failure.
Accuracy, speed and cost on the same 18 states
| Model | Correct action choices | Accuracy | Median API response time | P95 API response time | Actual API cost for 18 calls |
|---|---|---|---|---|---|
| Jev 1.13 | 17 / 18 | 94.4% | 0.38 s | 0.59 s | $0.000515928 |
| OpenAI Decisions · GPT-6 Luna | 18 / 18 | 100.0% | 0.45 s | 0.66 s | $0.000887000 |
| MS Decision-1 | 18 / 18 | 100.0% | 0.65 s | 1.00 s | $0.000253596 |
| Clef Flash | 14 / 18 | 77.8% | 0.59 s | 1.04 s | $0.000327256 |
| Solar Decide | 18 / 18 | 100.0% | 5.39 s | 14.99 s | $0.000927300 |
| Claude Haiku 4.5 | 14 / 18 | 77.8% | 1.01 s | 2.02 s | $0.024684000 |
| GPT-5.4 Mini | 13 / 18 | 72.2% | 1.19 s | 1.39 s | $0.007480500 |
We called every model through OpenRouter, ran one request at a time and rotated the model order for each state. Each got the same state, question and options; the decision-only models answered via the Decisions API, and Claude and GPT Mini via the same function-calling format as in the earlier experiment. We applied latency-first routing with provider fallback disabled, and excluded 1 warm-up per model. Max wait was 30 seconds, with no retries. None of the 126 responses used in the comparison had an API error or timeout. P95 is the 95th percentile of the measured response times, which shows how slow the slower responses got.
Cost is the sum of usage.cost from successful responses for these 18 states, in actual dollars. Each model handled the same states and questions 18 times, so totals don't differ because of survival time or call count. Warm-ups, the 19th state, the insufficient-credit response and the separate run stopped by a connection failure are excluded. Top-up fees, taxes and computer costs aren't included either. We sent inputs with the same meaning, but that doesn't mean the API formats and each model's token counting are the same.
To make the small amounts easier to compare, scaling the observed average per call to 1,000 calls gives $0.014089 for MS, $0.028663 for Jev and $0.049278 for OpenAI Decisions. These are proportional projections from these 18 calls, not the result of actually calling 1,000 times. The projections for every model are also in the CSV below.
The accuracy above is the share of answers that matched the predefined action rules, not a game survival rate. For example, Jev's one mismatch was choosing to duck in a situation where it could just keep running. That's an unnecessary move, but it doesn't mean a crash in itself. For the 6 states where jumping was correct, we also checked the jump type separately: Clef chose short vs. full jump differently on 1 of them, so only 13/18 of its responses matched on both action and jump type. Every other model's jump type matched the reference on all 6.
Download the same-state evaluation comparison table (CSV) · Download per-state inputs, answers and responses (CSV)
Solar was accurate, but we couldn't confirm the fast-choice advantage in this API setup
One of the core advantages of a decision model is returning a choice quickly without generating long text. The official Jev overview and the official Solar overview both emphasize this design. So alongside whether it picked the right answer, you have to check whether that answer arrived in the time you needed.
Given time to answer, Solar got all 18 shared states right. But its median API response time was 5.39 seconds, and in this API setup we couldn't confirm the fast-choice advantage that decision models advertise. Jev responded fast with a median of 0.38 seconds, but its action choices matched the rules on 17/18. OpenAI Decisions and MS Decision-1 got all 18 right in this sample, with medians of 0.45 and 0.65 seconds respectively.
These results can't be stretched into a general ranking of the models' reasoning ability. It's a small task with explicit rules, and each state was called only once. The API time also bundles communication, routing, server queuing and model compute together, so it isn't a measurement that isolates pure inference speed. What it does show is that decision accuracy and end-to-end service response time should be treated as separate metrics.
At work, too, you need to set those two criteria separately. If you're collecting inquiries to classify later, you can wait a few seconds; if you're choosing which urgent alerts to send right away, the correct answer has to arrive within a set time. You can only decide which model to use by checking accuracy on real work data and measuring acceptable response time and cost per call together.
On this task, MS came out ahead on cost and OpenAI on the balance of accuracy and speed
If we were to start the next evaluation of this task with one model, we'd look at MS Decision-1 first. Its action choices and jump types matched the reference on all 18/18, and its cost for 18 calls, $0.000253596, was the lowest of the seven models. Its median response time was 0.65 seconds. It was about 51% cheaper than Jev but slower to respond, so the choice depends on how much latency you can accept.
| Priority | Model to look at in this sample | What to check alongside |
|---|---|---|
| Follow the given rules exactly while keeping cost down | MS Decision-1 | 18/18, median 0.65 s. Check whether your real work can tolerate this latency |
| Keep everything correct with shorter response times | OpenAI Decisions | 18/18, median 0.45 s. About 3.5x the cost of MS on these 18 calls |
Jev was the fastest with a median of 0.38 seconds and was about 42% cheaper than OpenAI. But the median gap with OpenAI was 0.07 seconds, and its action choices matched the reference on 17/18. These results alone weren't enough to recommend Jev ahead of MS or OpenAI. You'd need to check further whether that small speed difference matters in real work and what effect a rule mismatch has.
Solar got 18/18 right but cost more and responded more slowly than MS and OpenAI, so there was little reason to pick it first for this task. Clef was cheaper than Jev, but its action choices were 14/18, and only 13/18 were right on jump type too. Claude and GPT Mini were also more expensive, slower and less consistent with the rules than MS and OpenAI on this task. That doesn't mean the ranking holds for other work like writing documents, coding or complex reasoning.
These selection criteria come from a small experiment that called the same 18 rule-defined states once each. We didn't combine accuracy, speed and cost into an arbitrary score to crown an overall winner. For real work, first decide what one wrong answer costs you and what response time you can accept, then check with more work samples and repeated calls. A model that got 18/18 this time isn't guaranteed to always get it right going forward.
How much did actual gameplay cost?
The amounts below aren't estimates from plugging input volume into token prices; they're the sum, in dollars, of usage.cost recorded in the successful API responses for each run. Responses requested during a run were included in its cost even if they arrived after the crash. Top-up fees, taxes and computer costs aren't included.
Actual cost of the single best run used in the Reel
| Round | Model | Survival time in game | API calls | Actual cost |
|---|---|---|---|---|
| 1 | Claude Haiku 4.5 | 15.38 s | 10 | $0.01368800 |
| 1 | GPT-5.4 Mini | 17.19 s | 13 | $0.00540075 |
| 1 | Jev 1.13 | 59.13 s | 59 | $0.00169180 |
| 2 | Jev 1.13 | 299.18 s | 329 | $0.00942858 |
| 2 | OpenAI Decisions | 299.04 s | 329 | $0.01619360 |
| 2 | Clef Flash | 45.55 s | 44 | $0.00044323 |
| 2 | MS Decision-1 | 25.18 s | 20 | $0.00028346 |
Round 1 Jev ended at 60 seconds of real time, and round 2 Jev and OpenAI Decisions at 300 seconds of real time. Because of things like the wait for the game to start, in-game time was about 59 and 299 seconds respectively.
The two full runs in round 2 had almost the same survival time and 329 calls each. In those two runs Jev cost $0.00942858 and OpenAI Decisions $0.01619360, so Jev was about 42% cheaper. That's a difference observed with this request format and course. Even when you send questions with the same meaning, API formats and token counts differ by model.
Normalizing for survival time: cost per minute of gameplay
Picking only the single best run can skew the results, so the table below adds up every valid run in each round, including runs that ended early. Round 1 has 7 runs each for Haiku and GPT Mini and 4 for Jev; round 2 has 5 each.
Cost per minute of gameplay = total cost of all valid runs ÷ total in-game survival time × 60
| Round | Model | Runs | Total survival time | Total actual cost | Cost per minute of gameplay |
|---|---|---|---|---|---|
| 1 | Claude Haiku 4.5 | 7 | 42.32 s | $0.02190200 | $0.031052 |
| 1 | GPT-5.4 Mini | 7 | 76.36 s | $0.01869975 | $0.014693 |
| 1 | Jev 1.13 | 4 | 236.38 s | $0.00676696 | $0.001718 |
| 2 | Jev 1.13 | 5 | 842.03 s | $0.02605218 | $0.001856 |
| 2 | OpenAI Decisions | 5 | 570.79 s | $0.02983160 | $0.003136 |
| 2 | Clef Flash | 5 | 112.18 s | $0.00096840 | $0.000518 |
| 2 | MS Decision-1 | 5 | 81.89 s | $0.00083614 | $0.000613 |
The per-minute figure isn't what it would cost to actually play the same 60 seconds in full. It's the spending rate observed while each model was alive, scaled to one minute. It also doesn't assume a model that crashed would come back and finish a full minute. As the game goes on, speed and obstacle density change, and requests sent right before a crash can finish late, so calls per minute vary.
At the observed spending rate in round 1, Jev was about 18x cheaper than Haiku and about 8.6x cheaper than GPT Mini. In round 2, Clef and MS had lower per-minute costs, but neither completed a 5-minute run. Ending cheaply and lasting long while staying cheap are two different things.
Normalizing by call count instead of time
Below is the cost from the same logs scaled to 1,000 successful calls. Each call includes both the action question and the jump-type question. Each API has a different input format, so this isn't a comparison of billing for identical token counts.
| Round | Model | API calls observed | Projected cost per 1,000 calls |
|---|---|---|---|
| 1 | Claude Haiku 4.5 | 16 | $1.368875 |
| 1 | GPT-5.4 Mini | 45 | $0.415550 |
| 1 | Jev 1.13 | 236 | $0.028674 |
| 2 | Jev 1.13 | 909 | $0.028660 |
| 2 | OpenAI Decisions | 606 | $0.049227 |
| 2 | Clef Flash | 92 | $0.010526 |
| 2 | MS Decision-1 | 59 | $0.014172 |
This tally only includes valid play by the models currently being compared in the two rounds. It excludes warm-ups, requests blocked for insufficient credits, excluded diagnostic runs, the earlier 60-second pilot experiments, the Solar experiment and the follow-up evaluation that gave the same states and waited. The 3 timeouts, where no response came back and the amount couldn't be confirmed, aren't in the totals either, so this isn't total account spend or total development cost. MS's confirmed amount including warm-up was $0.000850122; the table includes only the play cost of $0.000836136. Round 1 mixes default and latency-first routing, and MS was a follow-up experiment, so don't generalize this into a controlled unit-price ranking.
Download the cost comparison data (CSV)
So how do you connect this to your own work?
In the game, the model received the obstacle state and picked the next move. At work, too, you can start by testing repetitive tasks where you can narrow down the inputs and options.
| Task | Input to the model | Options to choose from |
|---|---|---|
| Classifying customer inquiries | Inquiry text and classification criteria | Billing · Refund · Technical |
| Deciding whether to alert | Event details and urgent-alert criteria | Alert now · Review in a batch |
| Choosing the next handler | Inquiry type and available handlers | Billing AI · Technical AI · Human review |
For example, when an inquiry comes in, the model picks the type and whether to alert, and code routes it to the right AI based on the result. You can set it up so a human only reviews inquiries that are exceptions or meet the review conditions you've defined. Surviving the dino game can't guarantee accuracy on this work. You need to measure classification accuracy, missed urgent inquiries, response latency and cost per call together, using evaluation data made of real inquiries labeled with correct answers.
The takeaway from this experiment for real work is to separate the steps that need a model that writes long explanations from the steps that need an API that quickly picks an option. How this was used in real public projects is collected in the Jev international use case roundup.
Surviving 60 seconds marks when the test ended, not game over
Every run was set to go for at most 60 seconds of real time. The selected Jev recording shows 59.13 seconds of game time; the two times differ slightly because of the wait for the game to start and when recording began. "Survived to 60 s · test ended" in the Reel means it hadn't crashed by that point, not that the game ended at 60 seconds.
Response time was measured from when the server sent the API request until it got the response. Costs in the tables are the dollar amounts recorded in usage for successful responses. Results change with run time, network and model provider status, and these recordings can't tell us when a crash would have happened in a longer run.
We separated GPT Mini's blocked first call from a real crash
When we first connected GPT-5.4 Mini, the first request got an HTTP 402. We had requested up to 128 tokens, but the response said the credits remaining at the time could only allow up to 116 tokens. That run had neither a model answer nor a crash, so we didn't count it as a result.
After updating the connection with a usable key, we kept the output limit at 128 tokens and reran the same course. This time we successfully got 13 API responses, and it crashed at 17.19 seconds of game time. We didn't slot the aborted earlier run in as a low survival time, and we didn't lower the output limit for GPT alone.
Keep the AI's choice separate from the code's timing
This experiment isn't evidence that Jev looked at the screen and played the game like a person. The three models share the code that reads obstacles and handles control timing, and a simple obstacle game like this can be played well by rule-based code alone. In real-time play, what we observed was whether the API's decision arrived in time with this connection setup and course. In the follow-up evaluation, we waited for the answers and checked agreement with the action rules separately. You need both results to tell apart a late answer from a different choice.
In the best-run highlight, the scenes after Claude and GPT Mini crash are shown as freeze frames, and Jev's continued play after they stopped is shown at 8x speed. If you try this yourself, it's better to record the same number of runs per model and compare full results and response-time distributions under the same routing conditions. Adding a baseline run that uses only game rules also makes the effect of adding a model clearer. If you're curious about real work applications, see the inquiry routing, feed filtering and task-path decision cases in the Jev use case roundup.
When you connect a decision model to your work, look at whether the right choice arrives within the time you need, and what that choice costs together. This time Solar got the answers right when we waited, but in the API setup we measured, we couldn't confirm the fast-choice advantage.
This experiment started with connecting Jev, but on this task, MS Decision-1 came out ahead on cost and OpenAI Decisions on the balance of accuracy and speed. Jev having the fastest responses isn't enough on its own to recommend it first. This isn't a model ranking for every kind of work; you should check again with answer data from your real work, and your acceptable response time and cost.