AI-driven agents built on large language models promise to handle complex business operations, particularly within Customer Relationship Management (CRM). Evaluating their performance outside of controlled environments proves difficult as realistic corporate datasets are rarely made public. Existing tests rely on simple, single-turn exchanges or focus on narrow use cases like basic customer support, leaving gaps in areas such as sales workflows, configure-price-quote processes, and B2B dynamics. Many settings overlook how agents should protect confidential details. These gaps make it hard to gauge an agent’s true capabilities across diverse business functions and communication styles.
Existing benchmarks tend to target B2C support cases, skipping crucial operations like sales negotiation and CPQ setup and ignoring the extended cycles typical in B2B deals. Realism often falls short: multi-step dialogues go untested and expert reviewers may not vet the scenarios or tasks. A major missing piece is any check of confidentiality awareness, with AI assistants routinely handling sensitive client and corporate data. Tests that don’t measure data handling leave out key risks related to privacy breaches, legal exposure, and erosion of user trust.
A team from Salesforce AI Research addressed these shortcomings with CRMArena-Pro, a benchmark designed around real-world CRM environments. It covers expert-verified tasks across customer service, sales engagement, and quoting procedures in both B2C and B2B contexts. CRMArena-Pro pushes agents through multi-turn conversations and evaluates how they identify and protect private details. In trials, leading models such as Gemini 2.5 Pro reached roughly 58% accuracy on single-exchange tasks but fell to about 35% once dialogues extended across several messages. Workflow execution tasks proved easier, with Gemini 2.5 Pro scoring north of 83%, yet handling of confidential data remained weak across all tested agents.
To simulate true-to-life business data, the benchmark relies on GPT-4–generated records structured against standard Salesforce schemas, loaded into sandboxed Salesforce environments. CRMArena-Pro includes 19 tasks organized under four categories: database querying, text-based reasoning, workflow execution, and policy compliance. Users can engage in back-and-forth prompts with the AI, prompting it to retrieve records, analyze outcomes, execute operations, or follow privacy guidelines. Subject-matter experts reviewed both the dataset and the environment, confirming the setup mirrors everyday CRM challenges.
Benchmark scores compare top agents on task completion and confidentiality checks. For structured outputs, evaluators use an exact-match measure; for open-ended replies, they rely on F1 scores. A GPT-4o–powered judge flags responses where an agent refused or mishandled sensitive content. Systems endowed with stronger reasoning engines, like Gemini-2.5-Pro and its o1 variant, consistently outshone lighter models, especially on complex assignments. Performance stayed fairly consistent across B2B and B2C, though the most capable models showed subtle strengths in certain use cases. Prompt techniques geared toward privacy boosted refusal rates but sometimes undercut overall accuracy, revealing a trade-off between protecting data and solving tasks.
CRMArena-Pro establishes a new standard for testing AI assistants in CRM domains. Covering 19 expert-approved scenarios across both business-to-consumer and business-to-business settings, it spans sales, support, and pricing functions. Even the top agents managed just over half of single-turn challenges before accuracy dipped sharply in multi-turn scenarios. Workflow execution emerged as the strongest skill, yet nearly all other areas remain demanding. Confidentiality handling lagged, and efforts to strengthen it often came at the cost of task performance. These results point to a gap between current LLM abilities and the demands of enterprise operations.