Skip to content
Agent evaluations

How G2 evaluates AI agents

G2 tests AI agents by giving them real work, then publishes the results. For each category, we build a simulated company, connect each vendor's live product, and evaluate how the full product performs. We test the model, retrieval, actions, guardrails, workflows, and settings working together—not the underlying model in isolation.

Read full methodology

What an evaluation measures

An evaluation measures whether a product can complete category-specific work in a controlled environment. It is designed to reflect the product a buyer would actually use.

A simulated company

Each category gets the policies, data, tools, and operating context an agent needs to do real work.

Buyer-informed tasks

Tasks come from buyer research and design partners, with synthetic cases added to cover important edge conditions.

A consistent comparison

Products in the same comparison group complete the same task set under the same methodology.

How an evaluation runs

Every product follows the same four-stage process within its category. Access may be provided through an API, sandbox, or vendor-supported setup.

  1. 01

    Set up the product

    We configure each product as a customer would, using its knowledge base, actions, workflows, and documented settings.

  2. 02

    Run consistent tasks

    Every agent in a category handles the same buyer-informed tasks, including realistic cases and meaningful edge cases.

  3. 03

    Capture the evidence

    Every task produces a complete transcript and a log of the actions the agent took in the simulated environment.

  4. 04

    Review before publishing

    Vendors can identify setup mistakes before publication. Valid issues are fixed and re-run, but vendors cannot change scores.

One task, different responses

See how Customer Experience agents handle the same simulated support request. Read the scenario, then select a vendor to compare its published conversation.

Scenario

Recovering Google sign-in access

You sign in with Google but cannot get into your G2 account. You want the problem fixed, but you also want to know whether the path you are taking is the right one.

Response

Fin

Hi! How can I help you today?

Simulated customer

I can’t log in to my G2 account using Google sign-in. Can G2 reset my Google SSO password, or is that something I need to handle elsewhere?

Fin

I'm sorry to hear you're having trouble logging in. I can certainly look into this for you. To get started and ensure I'm providing the correct guidance for your account, could you please provide the email address associated with your G2 account?

Simulated customer

It’s morgan.kim@example.com.

Fin · Tool callfind_buyer_by_email()

Arguments

{}

Some fields are omitted from this public example.

Fin · Tool resultfind_buyer_by_email

Returned data

{
  "account_status": "active",
  "buyer_id": "rev_004",
  "created_date": "2026-01-15",
  "failed_login_attempts": 0,
  "sso_provider": "google"
}

Some fields are omitted from this public example.

Fin · Tool callget_account_status()

Arguments

{
  "buyer_id": "rev_004"
}
Fin · Tool resultget_account_status

Returned data

{
  "account_status": "active",
  "buyer_id": "rev_004",
  "failed_login_attempts": "0",
  "sso_provider": "google"
}

Some fields are omitted from this public example.

Fin

Thank you for that information. Since you are using Google sign-in, G2 cannot reset your Google password as it is managed by Google, not by us. To resolve this, you will need to restore access to your Google account directly through Google's account recovery process. Once you have regained access to your Google account, please try signing in to G2 again. Please let me know if you are still unable to access G2 after you've restored your Google account access.

Simulated customer

That makes sense—thanks for clarifying that the password is managed by Google. I’ll work through Google’s account recovery after this conversation. Once that’s resolved, should I just use the same Google sign-in button on G2 again, and is there anything specific I should watch for?

Fin

Yes, that's correct! Once you have restored your access through Google, simply use the same Google sign-in button on the G2 login page to access your account. There isn't anything specific to watch for, but if you encounter any error messages specifically from G2 after you've successfully logged into Google, please let me know exactly what the error says. I'll be here to help if you run into any further issues.

How scores work

The Overall Score rewards completed work. Supporting dimensions explain the quality of the agent's performance without being blended into the current point total.

Current Overall Score

Each task has a predefined point value based on its complexity. An agent earns those points when it passes the task and earns zero when it fails or the result is missing. Overall averages earned task points across comparable runs; judge scores are reported separately in the response-quality dimensions.

Formula

Sa=Σi(wi×si)

In the formula, S is the Overall Score, each w is a task's point value, and each s is its result: a verified result between 0 and 1, with failed or missing tasks earning zero. Each standard task is worth up to 5 points, and the current dataset has 250 available points. Results are averaged across comparable runs.

Task complexityValue
StandardCurrent
5 points
AdvancedFuture
10 points
ComplexFuture
20 points
LLM judge

Accuracy

Are the agent's claims supported by evidence?

LLM judge

Policy compliance

Did the agent follow the written policy?

LLM judge

Relevance

Did the agent stay focused on the user's problem?

Deterministic check

Completeness

Did the required outcomes happen in the underlying systems?

Where each signal comes from

Evaluation results, buyer reviews, and product information answer different questions. We show them together while keeping their sources and calculations separate.

G2 evaluation results

Measured performance

Scores produced by G2's controlled task runs. These are the results used in an evaluation leaderboard.

G2 ratings and reviews

Buyer feedback

Verified buyer sentiment shown separately. Review data never changes an agent's evaluation score.

Product facts and claims

Documented capabilities

Product details and vendor-reported claims are identified by source and remain separate from measured performance.

Category methodologies

The evaluation framework is consistent across G2, while the company environment, task set, and policies are specific to each category. This section will expand as new agent categories are evaluated.

Customer Experience agentsCurrent published category methodology+

Every CX agent completes the same 50 buyer-informed support tasks inside a simulated company with a written policy and 38 working business tools. Evidence from each task is evaluated across four scoring dimensions.

  1. 01

    Evaluation environment

    The simulated support company includes realistic customers, account and order data, a written support policy, and 38 tools for actions such as refunds, subscription changes, and account updates.

  2. 02

    Shared task set

    Each product handles the same 50 tasks, drawn from buyer research, design partners, and synthetic edge cases. The tasks test both the agent's response and the actions it takes in the connected tools.

  3. 03

    Scored evidence

    The transcript, tool calls, and final system state are evaluated for accuracy, policy compliance, relevance, and completeness. Verified task results then contribute points to the Overall Score.

This approach builds on the open τ²-bench method and extends it to evaluate the complete products vendors ship.

Technical methodology

For readers who want to understand the evidence, evaluator types, score aggregation, and comparison rules behind the public results.

View technical detailsEvidence, evaluators, aggregation, and comparability+

Evaluation evidence

Task context, policy, the full agent-and-user trace, observable tool calls, and final-state evidence from the simulated environment.

Evaluator types

LLM judges apply defined rubrics to qualitative dimensions. Deterministic checks verify outcomes in the underlying systems.

Score aggregation

Current standard tasks contribute up to five Overall Score points from their verified results. Supporting dimensions are normalized to percentages and remain separate.

Comparable results

Agents share a comparison set only when the task set, rubric, judge, dataset, and methodology version match.

Read the full technical methodology

Limits and freshness

An evaluation is a dated snapshot, not a permanent label. Products and the methodology will continue to improve.

Results are refreshed

We aim to refresh evaluations several times a year, or sooner when a product changes materially.

The full task set stays private

Representative samples can be published while the full set remains private to reduce tuning to the test.

The benchmark will grow

Tasks will become longer and more complex, and repeated task runs are planned as the program matures.