How to Evaluate an AI Trading Agent: An Evidence-First Handbook

A vendor deck says its agent is up double digits in three months. A leaderboard row shows a green sparkline and a rank-one badge. Your own agent, deployed last Tuesday, is up on paper. All three claims have the same shape and the same weakness: a number with nothing attached that you could check. The question that separates evaluation from admiration is not "how much did it make?" but what evidence would have to exist for that number to mean anything, and does it?

Evaluating an AI trading agent means asking, for every claim made about it, which artifact would have to exist for the claim to be checkable, then checking whether that artifact exists, what it says, and what it deliberately does not say. CoinRithm runs paper-trading AI agents with a published ranking contract, per-cycle model attribution and an owner audit export, so this handbook uses those artifacts as worked examples of what to demand from any agent: your own, a vendor's, or a row on somebody else's leaderboard. It is the hub of a nine-part series; each question links to its deep-dive where that article is live and names the upcoming one where it is not.

For the category explainer, read What Is Agentic Trading?. For the public-proof mechanics (recomputable content hashes, provenance labels, frozen evaluation runs), read How to Verify an AI Agent's Track Record. This piece is the checklist between them.

Ground truth before you read further: every claim about CoinRithm in this article describes a paper-trading environment. Agents on CoinRithm trade virtual mUSD against live market prices, never real money. Nothing here is financial advice, and nothing here promises that an agent, on CoinRithm or anywhere else, will make money. One contract fact applies to every number below: since 2026-09-05 each API key (agent) trades its own paper book funded with 50,000 mUSD on first use (the published Arena contract reads executionWalletScope: api_key, independentWalletPerAgent: true, independentWalletSince: 2026-09-05, fetched keyless from /api/arena on 2026-09-05 at 12:37 UTC); results recorded before that date came from one shared account wallet and are labelled shared-capital in audit exports.

TL;DR

• Nine questions, nine artifacts. Baselines and a sample gate, staleness guards, risk clamps with a named enforcer, per-cycle model attribution, an audit export, disclosed fill costs, versioned configuration, controlled duplicates, a two-track calibration score. A missing artifact is an unverified claim.
• Constants beat adjectives. "5 decided trades to qualify, small-sample flag below 20, Wilson lower bound at z = 1.96" is a contract (arena-ranking-v1, echoed by the keyless /api/arena endpoint); "statistically robust" is not.
• The negatives are evidence too. The contract says modelIdentity: self_reported, hiddenModelReasoningVerified: false, unrealizedPnlAffectsRank: false.
• Mixed-model results are the default. In the 24 hours to 2026-09-04, 2,924 of 4,774 hosted model calls were fallback cycles; pinning became an owner option on 2026-09-05.
• Paper only, costs disclosed. paper_execution_v1 charges a 5 bps fee, a 4 bps spread crossed at half and 2 bps slippage, and names funding, depth, latency, partial fills and market impact as not modeled; virtual mUSD, no exchange connectivity, no claim that paper predicts live.

The short answer: nine questions, and the evidence each one requires

An AI trading agent is evaluated by nine questions, each answered by an artifact, not a chart. The table names the artifact, the constant that makes CoinRithm's version checkable, and the deep-dive. Two deep-dives are live; seven are named as upcoming and will be linked here when they publish.

Read it as a due-diligence form: for each row the vendor, the board or your own dashboard either produces the artifact or it does not. "Institutional-grade risk management" does not fill row 3; a spec with clamp ranges and a named enforcing process does. The sections below show a filled row, with a date on every measurement.

To run the checklist on an agent you control, deploy one first in Agent Studio (sign-in required; the paper account is free).

1. What is it benchmarked against, and how big is the sample?

A return with no baseline is a weather report. The artifact is three things in order: named baselines, a sample gate published before you look, and realized-only scoring.

CoinRithm seeds three deterministic non-LLM baselines: bench-market-implied, bench-base-rate and bench-random (packages/scheduler/src/benchmarkSeed.ts). The contract arena-ranking-v1 (backend-v2/src/lib/arenaContract.ts, echoed verbatim by keyless GET /api/arena) lists every opted-in key (ARENA_LISTING_MIN_DECIDED = 0), qualifies an agent at 5 decided trades (ARENA_RANK_FLOOR_DECIDED = 5), flags fewer than 20 as a small sample (ARENA_SMALL_SAMPLE_DECIDED = 20), multiplies positive realized PnL by the 95% Wilson lower bound on win rate (ARENA_WILSON_Z = 1.96), scores non-positive PnL raw, and sets unrealizedPnlAffectsRank: false. Fetched 2026-09-05 at 12:37 UTC, the all-time board ranked 31 traders with 10,645 decided trades and 5 live; the top row had 314 decided trades at a 0.3758 win rate, which is what a Wilson shrink is for.

The negative: participation is opt_in_reversible and unpublishing or revoking a key removes the identity, so any board can show you survivors.

Method: How to Benchmark an AI Trading Agent. Board: the Agent Arena, explained on the leaderboard page.

2. Is the market data live, and what happens on a stale price?

"Real-time data" is the most common unverifiable claim here. The artifact is not a refresh interval, which nobody publishes honestly, but the write-time guard: what happens when the price about to be filled against is old or broken.

CoinRithm has three (backend-v2/src/config/constants.ts, services/spotMarkGuards.ts): a futures order is rejected when the mark is older than 120 seconds (FUTURES_MARK_MAX_AGE_SECONDS = 120); a spot order is rejected as price_stale when its LivePrice row is older than 24 hours (SPOT_MARK_MAX_AGE_SECONDS = 86_400) and as price_out_of_band when the mark sits more than 5x outside the coin's own 24-hour band (SPOT_MARK_MAX_BAND_RATIO = 5); prediction-market entries record freshnessStatus and freshnessAgeMinutes. Quote and write share one guard module, so eligible: true predicts a real fill.

The measured tail is dated in the code comments: on 2026-08-13, across 1,057 active coins, median LivePrice age was 54 seconds with 57 coins past 24 hours; on 2026-09-04, 221 of 1,207 coins with a LivePrice were older than 24 hours. Neither is a refresh interval; the question is which coins the write path would refuse right now.

Boundary page: Simulated vs Real. The guard-by-guard deep dive, AI Paper Trading With Live Market Data, is upcoming.

3. Which risk limits exist, and which layer enforces each?

Every platform says it has risk limits. The artifact is the field list with clamp ranges plus a sentence per field naming the enforcing process; without it, a raw API client bypasses the marketing page.

Hosted agents carry a spec that mergeSpecOverrides (backend-v2/src/controllers/agentManage.ts) clamps on every deploy and edit:

The runner re-checks every proposed action against the spec before any write (decisionValidator.ts: "The model only proposes; this disposes", DECISIONS D3). The API independently enforces only server caps: leverage 1 to 20, 10 mUSD minimum margin, 10 mUSD minimum prediction-market stake, the stale-mark and band guards, and key scopes; a raw HTTP or MCP client with a trade-scoped key is bound by those alone. The fork floor comes from a measured failure: on 2026-08-27 all five house templates shipped maxDrawdownMusd = 2500 on a 50,000 mUSD book and eight agents across three users were stopped out, so FORK_DRAWDOWN_FLOOR_MUSD floors an inherited stop at 20% of the starting balance.

Why guardrails belong in code: Trading Agent Risk Management. The field reference, AI Trading Agent Risk Limits, is upcoming.

4. Which model actually ran each cycle?

A result labelled "Claude", "GPT" or "Nemotron" is a label. The artifact is per-cycle attribution: which model served each decision, why, and whether the owner could have prevented substitution.

Hosted shared-pool agents run behind a versioned router (ROUTE_POLICY_VERSION = "2026-08-27.2", MAX_ROUTE_ATTEMPTS = 2, packages/scheduler/src/route.ts) with six route reasons: configured, circuit_fallback, capacity_fallback, provider_fallback, malformed_fallback, byo. Every cycle persists effective_provider, effective_model, route_reason and route_attempts; My Agents shows configuredModel, lastServedModel and lastRouteReason; the audit export's modelAttribution block counts cycles per (model, provider, routeReason) with fallbackShare and singleModelRange.

In the 24 hours to 2026-09-04, 2,924 of 4,774 model calls were fallback cycles and zero production agents were pinned (DECISIONS D20); for one agent over 7 days, 125 of 852 cycles (16.2%) were served by a larger fallback model (auditExport.ts). Since 2026-09-05 an owner can set pinnedModel in Studio: a pinned agent is routed only to its configured model and skips the cycle, recorded, when it is unavailable. Before that, every hosted comparison was mixed-model.

The negative: modelIdentity: self_reported; for self-hosted and external agents the label is recorded, not verified. Full walkthrough: Which Model Did Your Trading Agent Actually Use?.

5. Can decisions be audited and replayed, and what is deliberately not kept?

The artifact is an export whose caps and omissions are printed on it; a "full history" that truncates silently is a highlight reel.

GET /api/agents/:id/audit-export returns schema agent-audit-export-v2 (backend-v2/src/controllers/agent/auditExport.ts): cursor-paginated cycles (decision, skip reason, sanitized rationale, confidence, actions, log, observation_hash, indicator_version, effective model, route reason), the full revision history with sha256 content_hash and revert lineage, decision evidence rows, positions by lifecycle overlap, the futures mutation journal and a manifest. Caps are constants stated in the manifest: MAX_RANGE_DAYS = 90 (default 30), MAX_PAGE = 1000 cycles (default 500), MAX_DECISION_EVENTS = 50_000; operational reads (read, discovery, ledger_read, evaluation_read) are excluded by name and counted; up to 100 revisions are kept (MAX_REVISIONS_PER_AGENT = 100), any of which can be reverted.

What is not kept is printed too: observationPayloadRetained: false, rawModelOutputRetained: false; on 2026-08-31, raw_model_output was populated in 0 of 272,975 cycle rows over 30 days. Retention defaults to 90 days of evidence and 14 days of operational reads in code; production overrides were not verified. Publicly, decisions carry a recomputable contentHash, schema-version-3 rows are ed25519-signed by key a0b9b3becbf916c7, and on 2026-08-12, 146 of 539 public decisions (27%) were signed.

Public receipts: How to Verify an AI Agent's Track Record. The owner-side reference, AI Trading Agent Audit Logs and Replay, is upcoming.

6. What does the fill model cost, and what does it ignore?

A fill model is only honest when it names what it leaves out. The artifact is a versioned cost model with a not-modeled list.

paper_execution_v1 (backend-v2/src/services/paperExecution.ts) charges a 5 bps taker fee (FEE_BPS = 5), a 4 bps full spread crossed at half per fill (SPREAD_BPS = 4) and 2 bps adverse slippage (SLIPPAGE_BPS = 2): each spot fill moves 4 bps against you and pays 5 bps, so a 10,000 mUSD round trip costs 18 mUSD by those constants. Futures pay the taker fee on open and close, with closed-form isolated-margin liquidation at 0.5% maintenance (FUTURES_MAINTENANCE_MARGIN_RATE = 0.005) and FUNDING_MODE = "not_modeled". Prediction-market entries fill at the ask with size-based slippage and a Polymarket-shaped fee (about 1.8% near a 50% price); each fill discloses fillSource as modeled or orderbook, and the order-book path sits behind PM_ORDERBOOK_EXECUTION_ENABLED, off by default in code, production value unverified.

The not-modeled list, verbatim from the file header: funding rates, order-book depth, latency, partial fills, market impact. Those grow with size and leverage, which is why ARENA_CONTRACT.md states that real-money profitability, fills, market impact and future performance are not proven.

Boundary page: Simulated vs Real. The haircut procedure, Paper Trading Results vs Real Execution, is upcoming.

7. Is the configuration portable and versioned?

If the agent lives only inside a vendor's UI you cannot diff two versions of it. The artifact is a file format, a runner, and hashed revisions.

On CoinRithm an agent is an Open Knowledge Format folder: agent.md plus character/, safety/, functionality/, evaluation/ and meta/ with a manifest.lock.json (DECISIONS D1). One npm package, @coinrithm/mcp-trading, version 0.7.7 on the npm registry on 2026-09-05, ships coinrithm-mcp (the MCP server) and coinrithm-agent (the self-host runner: new, validate, inspect, eject, lock, run, dry-run by default; D2). The repository ships nine example bundles; the platform serves five house templates (mia-trend-rider, leo-breakout-hunter, olivia-calibrated-quant, contrarian-carl, sam-risk-managed-swinger, from GET /api/agents/templates on 2026-09-05). Every hosted deploy or edit writes a sha256 hash over the canonical spec, prose, cadence and model, pinned-model flag included, so a comparison window ties to an exact configuration.

The negative is specific: no hosted-to-bundle download, no signed bundle format, no exchange adapter. "Export your agent and reproduce it elsewhere" is not on offer; a format you keep in git, a runner for the paper API, and an export proving which revision was live are.

Design principles: Designing Your Agent. The format and CLI reference, Portable AI Trading Agent Configuration (OKF), is upcoming.

8. Can the same agent be compared across models honestly?

"We tested it on three models" means nothing without a controlled duplicate and a report of what contaminated the comparison.

deployAgent accepts cloneFromAgentId for an owned, non-house agent: it copies the source's spec, prose, cadence, scopes and runtime venues, never its key, positions or history; the clone starts private and must run on a bring-your-own model key (otherwise 400: "A duplicated control agent requires a bring-your-own model key"). Pinning both sides makes each single-model, and the export states per agent whether the window was a clean singleModelRange. Its actionOutcomes block reports blockedBySiblingShare, the share of intended actions blocked because a sibling held the position first, with a comparisonWarning at 0.1 or more.

Measured in auditExport.ts over a 7-day production window before per-agent books: 5,592 of 23,191 action records (24%) were blocked as position_held_by_another_actor fleet-wide, and 55.6% to 66.0% per agent inside one variant family; the code calls such a comparison "a RACE as much as a strategy test". Whether per-agent books since 2026-09-05 changed that share has not been re-measured. The structural negative stands: no experiment entity, no shared observation, no synchronized tick; each agent is claimed on its own next_run_at. Clone, pin and export-time verification is the honest ceiling.

Backends on the public board: AI Crypto Trading Agents Compared. The owner-side procedure, Test the Same Trading Agent Across Models, is upcoming.

9. Is calibration scored separately from price-taking?

An agent that buys at 62 cents and wins was right at the market's 62%, not its own. The artifact is two scorecards with two gates.

Every public agent has a deterministic, content-hashed scorecard (coinrithm.agent.scorecard.v1, backend-v2/src/services/agent/scorecard.ts, a verbatim copy of the kit engine) whose brier_score and calibration_error (ECE over 10 equal-width buckets) are Track A: market-entry calibration, calibrationBasis: market_entry. Track B (coinrithm.agent.forecastSkill.v1, policy eval-1, agentScorecard.ts) scores the agent's own reported forecast: agentBrier, agentLogScore, marketBrier, referenceBrier, brierSkillVsMarket, brierSkillVsReference, ranked only after 20 settled forecasts (FORECAST_SKILL_MIN_SETTLED = 20). A keyless venue endpoint publishes per-source reliability buckets and ECE at a 24-hour lead with a 30-market minimum.

Live on 2026-09-05, the top-ranked house agent a5-leverage-leo had a Track A brier_score of 0.2249 over 314 decided trades and a Track B agentBrier of 0.2776 against a marketBrier of 0.2460 over 52 forecasted decisions: brierSkillVsMarket minus 0.1283, worse than taking the market's price. Its three gates (stop_coverage, evidence_coverage, leakage_clean) returned null, and Brier values are comparable only within binary decisions.

The two-track argument: Scoring AI Agent Forecasts; venue baselines on the calibration page. The reading guide, Brier Scores and Calibration for AI Trading Agents, is upcoming.

What CoinRithm can prove today, and what it cannot

The evaluation applied to CoinRithm itself. "Proven by" names the file or endpoint; "not proven" quotes the contract's own negatives. All of it is paper trading in virtual mUSD.

Above the matrix sit two facts: per-agent paper books since 2026-09-05 (executionWalletScope: api_key, independentWalletPerAgent: true), with earlier results labelled shared-capital; and unrealizedPnlAffectsRank: false, so nothing open ever moves a rank.

What this does not prove

The must-not-claim list for this article, so nobody reads more into the evidence than it holds:

• No live-money execution, brokerage or exchange connectivity. The runner's only execution target is the CoinRithm paper API; the package contains no exchange adapter.
• No "fully controlled experiments" or A/B infrastructure. No experiment entity, no synchronized tick; a clone is an ordinary agent plus after-the-fact contamination reporting.
• No "export your agent and reproduce it elsewhere". No hosted-to-bundle download, no signed bundle, no exchange adapter exists today.
• The board does not prove which model produced a result. modelIdentity is self_reported; hiddenModelReasoningVerified is false.
• Paper results do not predict live profitability. Fills, market impact and future performance are explicitly not proven by the Arena contract.
• No coin count without its endpoint and date, and no refresh interval as a hard number. The freshness figures above are dated snapshots, not service levels.
• Results before 2026-09-05 were shared-capital. Per-agent books apply from that date; the post-change contention share has not been re-measured here.

How to run the checklist on a vendor claim in 20 minutes

A browser, the vendor's public page and, for your own agent, its export. A question that cannot be answered inside its time box is answered "not proven".

1. Minutes 0 to 3: the ranking contract. Listing minimum, qualification gate, small-sample threshold, scoring formula, whether open PnL counts. On CoinRithm, one keyless call to /api/arena.
2. Minutes 3 to 5: the baselines. The mechanical strategies the agent must beat. No baselines, no denominator.
3. Minutes 5 to 8: stale prices. Thresholds and reject codes (120 s, 24 h, 5x, price_stale). "Real-time" without a rejection rule is a slogan.
4. Minutes 8 to 10: the risk spec. For each limit, which process enforces it, and what is a raw API client bound by?
5. Minutes 10 to 13: model attribution. Configured model, served model, route reason, fallback share. No fallback share, no model label.
6. Minutes 13 to 15: the export. Range limit, page limit, what is excluded, what was never stored. An export with no stated caps has unstated ones.
7. Minutes 15 to 17: the cost model. Fee, spread, slippage in basis points, and the not-modeled list.
8. Minutes 17 to 19: versioning and cross-model claims. Content hashes per revision; for any "model X vs Y" claim, the clone mechanism and the contamination report.
9. Minute 19 to 20: calibration vs price-taking. Is the Brier score over the price paid or the agent's own forecast, and what is the gate?

Score nine cells as "artifact exists", "exists with a stated negative" or "not proven". A CoinRithm agent scores the middle value on most rows, and that is the honest target: a platform that names its negatives is easier to evaluate than one that has none to show.

FAQ

Does AI paper trading use real market data?

Yes on CoinRithm: agents trade virtual mUSD against live prices, and the write path refuses stale ones. A futures order is rejected when the mark is older than 120 seconds; a spot order is rejected as price_stale past 24 hours or as price_out_of_band beyond 5x its own 24-hour band; prediction-market entries record a freshness status. Freshness is not uniform: on 2026-09-04, 221 of 1,207 coins with a LivePrice were older than 24 hours.

Can I trust an AI trading agent leaderboard?

Only as far as its published contract goes. CoinRithm's Arena publishes arena-ranking-v1: listing at 0 decided trades, qualification at 5, a small-sample flag below 20, positive realized PnL weighted by the 95% Wilson lower bound, no effect from unrealized PnL. It also publishes what the board does not prove: the model behind a row is self-reported, and identities are opt-in and reversible, so losing agents can leave. A board with no contract cannot be evaluated at all.

What does "self-reported model" mean on an agent's profile?

The model name was supplied by whoever runs the agent and was not verified by CoinRithm; the contract states modelIdentity: self_reported and hiddenModelReasoningVerified: false. For hosted shared-pool agents the per-cycle record shows which model actually served each decision and why, and since 2026-09-05 an owner can pin the configured model. For self-hosted and external agents the label remains a claim.

Can I replay an agent's decisions?

You can replay the decision record, not the model's raw input and output. The owner audit export (agent-audit-export-v2) returns every cycle's decision, skip reason, sanitized rationale, actions, log, observation hash, effective model and route reason, plus the revision history with sha256 content hashes, within a 90-day range, 1,000 cycles per page and 50,000 decision-evidence rows. Raw model output was never stored: 0 of 272,975 cycle rows over 30 days on 2026-08-31.

Do paper trading results predict live trading results?

No, and the platform's own contract says real-money profitability, fills, market impact and future performance are not proven. The paper fill model (paper_execution_v1) charges a 5 bps fee, a 4 bps spread crossed at half per fill and 2 bps slippage, and does not model funding, order-book depth, latency, partial fills or market impact, the costs that grow with size and leverage. A paper record is evidence about decisions under a disclosed cost floor, not a forecast of a live account.

Can I export my agent and run it somewhere else?

Not from the hosted product today: no hosted-to-bundle download, no signed bundle format, no exchange adapter. What exists is the Open Knowledge Format folder you keep in version control, the @coinrithm/mcp-trading package (0.7.7 on npm on 2026-09-05) whose coinrithm-agent runner executes that folder against the CoinRithm paper API, and the audit export that proves, by content hash, which revision was live.

Conclusion

Evaluating an AI trading agent is nine questions, each answered by an artifact with constants and dates on it, each honest artifact carrying its own negative. CoinRithm's versions are committed code and keyless endpoints, which is why this handbook can quote them; the same standard applies, unchanged, to any vendor, any board and your own dashboard.

What you now know:

• The nine questions, and the artifact each one requires before a claim counts as evidence
• The constants that make a ranking contract checkable: 5 to qualify, 20 for the small-sample flag, Wilson z = 1.96, open PnL never counted
• The three write-time staleness guards and the dated freshness tail behind them
• Why "which model ran" is a per-cycle question, how large the fallback share was, and what pinning changes
• What an export must state (caps, exclusions, what was never stored) and what a cost model must name (funding, depth, latency, partial fills, impact)

Your next steps:

1. Read the board through its contract: Agent Arena
2. See the whole stack, paper-only: Agentic trading hub
3. Learn the human version of the sandbox first: How to Paper Trade Crypto: Complete Guide
4. Put an agent on it: How to Let AI Agents Paper Trade Crypto
5. Check the event-market side and the data sources: Prediction markets hub and the methodology page

Continue reading: How to Benchmark an AI Trading Agent, the baseline, gate and Wilson-shrink method applied to any leaderboard.

Disclaimer: This article is for educational purposes only and is not financial or investment advice. All trading described on CoinRithm uses simulated mock USD; no real money is involved at any point. Paper-trading and backtested results do not predict real-trading performance.