ATHENA | Markopolo AI
BENCHMARK RESULT
First against every frontier model we tested.
Athena is not a larger generalist. It is a specialist trained on what real shoppers actually do next — fine-tuned for the task, compared against frontier models prompted zero-shot. That is the intended comparison, and on this benchmark, the specialist wins.
Results Summary
- 24.50% exact match, first place
- 4B parameters, dense and self-hostable
- 992 actions in our full test harness
- 32K token training context
- 1 epoch of LoRA fine-tuning
OPeRA next-action leaderboard
Strict exact match on the full official 992-action test set
| Model | Type | Score | Margin |
|---|---|---|---|
| Athena mvrko-sim-1 | fine-tuned specialist | 24.50% | No. 1 |
| GPT-5.6 | frontier | 22.58% | -1.92 |
| GPT-4.1 | frontier | 21.50% | -3.00 |
| Claude Sonnet 5 | frontier | 18.35% | -6.15 |
| Claude Opus 4.8 | frontier | 12.70% | -11.80 |
How strong is each margin?
| Model | Exact-match points relative to Athena |
|---|---|
| GPT-5.6 | +1.92 nominal edge |
| GPT-4.1 baseline | +3.00 clear |
| Claude Sonnet 5 | +6.15 separated |
| Claude Opus 4.8 | +11.80 decisive |
What real shoppers click
Percent of click actions. All 5,051 click actions in OPeRA-filtered.
| Action Type | Count | Percent |
|---|---|---|
| review | 1,052 | 20.8% |
| search | 763 | 15.1% |
| product_option | 700 | 13.9% |
| product_link | 537 | 10.6% |
| other | 449 | 8.9% |
| purchase | 321 | 6.4% |
| nav_bar | 283 | 5.6% |
| page_related | 198 | 3.9% |
| quantity | 191 | 3.8% |
| suggested_term | 182 | 3.6% |
| cart_side_bar | 145 | 2.9% |
| cart_page_select | 139 | 2.8% |
| filter | 91 | 1.8% |
Real shopping behavior is mostly deliberation. Reviews, search, product options, and product links account for 60.4% of clicks. Purchase-intention clicks are only 6.4%.
THE BENCHMARK / ERROR ANALYSIS
Real people, real sessions, exact targets.
OPeRA is a public dataset of real people shopping online, recorded over four weeks. The challenge: given the page, what the shopper has done so far, and what they came to buy — predict their exact next move. Close doesn't count.
- 51 real shoppers
- 692 full shopping sessions
- 28,904 full action-observation pairs
- 5,856 filtered benchmark pairs
- 992 actions in OPeRA-test
Example Action
{
"action_type": "click",
"click_type": "product_link",
"semantic_id": "active_item_list.<product>.product_detail"
}
The frontier knows it should click. It clicks the wrong thing.
Wrong-button predictions
Share of all examples — published OPeRA error analysis, range 60.8% to 75.8%.
| Model | Wrong-button predictions |
|---|---|
| GPT-4.1 | 75.8% |
| DeepSeek-R1 | ~71% |
| Claude 3.7 Sonnet | ~66% |
| Llama 3.3 70B | 60.8% |
ARCHITECTURE AND TRAINING / EFFICIENCY
A specialist built around the structure of behavior.
Athena is a LoRA fine-tune of an open 4B base. The result comes from turning a live interface into a compact, typed observation, preserving the full journey at 32K context, and supervising directly on the action a real shopper took.
Training Recipe
- OBSERVE: Live page and session - Rendered page state, interaction history, current goal, and the named elements a shopper can act on.
- REDUCE: Structure-preserving parser - Discard styling and irrelevant markup. Keep actionable elements, hierarchy, function, and stable semantic IDs.
- MODEL: Athena, 4B - Long-context behavioral specialization over a 32,768-token training window.
- ACT: Structured next event with deterministic decoding.
- Base model: Built on Qwen3-4B — an open model with 4 billion parameters, small enough to run on your own hardware.
- Fine-tuning: Lightweight adaptation (LoRA) — we teach the existing model shopper behavior instead of training one from scratch.
- Memory: Reads up to 32,768 tokens at once — the shopper's whole session plus a simplified view of the page.
- What it learns: Given everything so far, what does this shopper do next.
- Training time: Single pass over the data (1 epoch) — no exotic compute required.
- Answers: Always gives the same answer for the same input, graded only on exact matches.
FROM PREDICTION TO PRODUCT
One event model, many customer surfaces.
The broader mvrko program applies the same pattern everywhere: observe a state, estimate what comes next, choose an intervention with a measurable outcome.
INDUSTRIES
| Industry | Description |
|---|---|
| FINTECH AND CYBERSECURITY | Detect malicious behavior before the transaction. |
| Fraud leaves a behavioral trace before it leaves a payment trace. | |
| CUSTOMER ENGAGEMENT | Give customer-facing agents a next-best-action layer. |
| AGENTIC COMMERCE | Construct the winning offer in one round trip. |
| CONSUMER BUSINESS | See churn forming before the cancel click. |
| RETAIL | Rank the interface by predicted shopper behavior. |
| MOBILE APP ECOSYSTEM | Predict the drop-off step and the right return window. |
| GAMING ECOSYSTEM | Read the session arc before frustration becomes a quit. |
| B2B | Turn a long account trail into the next commercial move. |
| REAL ESTATE AND PROPTECH | Separate listing traffic from a forming purchase journey. |
| INSURANCE | Catch quote abandonment and anomalous claims paths. |
Limitations of the Model
- Fine-tuned versus prompted: Athena is a fine-tuned specialist; the frontier models were prompted zero-shot.
- Observation format: Athena expects the exact OPeRA observation schema it was trained on.
- The GPT-5.6 margin is nominal; narrow enough to read as parity at the frontier.
- Exact match is deliberately unforgiving; scoring counts a prediction only when it matches the ground-truth action exactly.
- OPeRA is a simplified shopping benchmark that captures a constrained slice of shopping behavior.
- Domain scope: Athena is tuned for e-commerce shopper behavior.
- Prediction is not authority: High-stakes uses in finance, security, insurance, or employment should keep people in the decision loop.
Conclusion
Athena is the first release in the mvrko simulation track: a compact behavioral foundation for next-action prediction, journey simulation, and agentic decision systems.