2026-06-01 · 15 pages
Buying an agent platform in 2026 is harder than it should be. Every vendor pitches "agents," "memory," and "integrations." Few are comparable along the dimensions that decide success in production. This paper proposes a 12-criterion rubric and shows how to apply it.
The criteria, ranked by weight:
1. Authoring speed for non-engineers (15%). Time from "I want an agent that does X" to a working first version. Measure in seconds, not in tutorial completion rate.
2. Cost predictability and kill switches (12%). Is there a workspace-level hard cap on spend per day? Per agent? Per tool? Without a kill switch, every team eventually has a runaway loop.
3. Per-user credential isolation (10%). Are end-user credentials encrypted, scoped, and revocable? Is row-level security enforced? Are credentials excluded from logs?
4. Callability over MCP (10%). Can other planners call this agent as a tool? Is the catalog discoverable? Are the tools described in a standard schema?
5. Persistent per-agent memory (8%). Does the agent remember between runs? Can the operator read and edit that memory? Is it portable?
6. Real OAuth integrations vs DIY (8%). Are the OAuth flows guided, with refresh rotation handled? Or are operators wiring tokens manually?
7. Trigger surface (8%). Manual, schedule, webhook, agent-to-agent. Missing any of the four is a real gap.
8. Runtime observability (7%). Can the operator watch an agent think? Replay a failed run? Inspect intermediate tool calls? Without this, debugging is impossible.
9. Multi-model routing (5%). Is the platform locked to one model provider? Can it route to a cheaper model for narrow steps?
10. Approval gating (5%). Can destructive operations require human approval inline?
11. Onboarding cost for new teammates (5%). How long until a new operator on the team can ship an agent? Hours or days?
12. Vendor independence (7%). What happens if you stop paying? Can you export agent definitions, memory files, and run history?
The paper applies the rubric to GO Pilot GO and to a representative sample of competitors (Claude sub-agents, ChatGPT custom GPTs, Zapier, n8n, Lindy, Relevance AI, CrewAI, LangGraph, Retool Agents, OpenAI Assistants API), with weighted scores. We do not pretend the weights are universal; the rubric is the contribution, the scores are illustrative.
The conclusion: most teams underweight cost predictability and credential isolation, and overweight integration count. The first two failures cost real money and trust; integration count is mostly a marketing artifact in the MCP era.