Open vs Frontier
Fraud Detection: Two Open Models vs Frontier
Fine-tuned Llama 3.1 8B (US open weights) and Qwen 2.5-7B (Chinese open weights), both post-trained on Oumi, score the same synthetic guest-checkout transactions as Claude Sonnet. Inspired by Coinbase's fraud-agent work, this is a separate experiment, not a reproduction of Coinbase's proprietary benchmark.
Block at risk level 4 or 5. Scores are checked against known outcomes. The 300-transaction figures are reported separately below; the interactive run replays 30 recorded synthetic outputs. No live model calls are made.
Reported 300-transaction benchmark
| Model | Fraud caught | Fraud value caught | Precision | F1 | Cost / 100k |
|---|---|---|---|---|---|
| Llama 3.1 8B | 100% | 100% | 98.5% | 99.2% | ~$4 |
| Qwen 2.5-7B | 100% | 100% | 98.5% | 99.2% | ~$4 |
| Claude Sonnet | 100% | 100% | 98.5% | 99.2% | ~$130 |
Figures reported in the supplied experiment summary. The replay below contains 30 recorded synthetic transactions, not the full test. Open-model costs use an estimated blended compute rate; Claude costs use recorded API usage. These figures imply about a 30× inference cost gap. They don't include training or total operating costs.
0 / 30 transactions
Llama 3.1 8B
Meta · US open weights · fine-tuned on Oumi
- Fraud caught
- —
- Precision
- —
- Cost
- $0.000000
Qwen 2.5-7B
Alibaba · Chinese open weights · fine-tuned on Oumi
- Fraud caught
- —
- Precision
- —
- Cost
- $0.000000
Claude Sonnet
Frontier model · external API
- Fraud caught
- —
- Precision
- —
- Cost
- $0.000000
Cost so far
Recorded replay
This interactive replay needs JavaScript. It compares 30 recorded model outputs and makes no live model calls.