.NET: Bug: FoundryEvals sends `azure_ai_evaluator` testing-criteria type — HTTP 400 from current Foundry endpoints

[joslat]

Summary

FoundryEvals.EvaluateAsync always returns HTTP 400 against current Azure AI Foundry project endpoints (*.services.ai.azure.com/api/projects/*). The service rejects the request body with:

HTTP 400 (invalid_request_error)
Invalid set of testing criteria. Invalid value: 'azure_ai_evaluator'.
Supported values are: [label_model, text_similarity, string_check, label_model, score_model, python, endpoint]

Root Cause

In dotnet/src/Microsoft.Agents.AI.Foundry/Evaluation/FoundryEvalWireModels.cs, the WireTestingCriterion type defaults to "azure_ai_evaluator":

internal sealed class WireTestingCriterion
{
    [JsonPropertyName("type")]
    public string Type { get; init; } = "azure_ai_evaluator";   // ← rejected by current API
    ...
}

The current Foundry Evals REST API at *.services.ai.azure.com no longer accepts azure_ai_evaluator as a criterion type. The bug is present in the current main branch and in the published nightly 1.12.0-preview.260629.1.

Repro

// Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1
// Azure.AI.Projects 2.1.0-beta.3
// Endpoint: https://<hub>.services.ai.azure.com/api/projects/<project>

AIProjectClient client = new(new Uri(endpoint), new DefaultAzureCredential());
FoundryEvals evals = new(client, "gpt-4o", FoundryEvals.Relevance, FoundryEvals.TaskAdherence);

// Throws ClientResultException: HTTP 400 (invalid_request_error)
await agent.EvaluateAsync(queries, evals);

The Azure AD authentication succeeds. The failure is in the POST to CreateEvaluationAsync where the request body contains "type": "azure_ai_evaluator" in testing_criteria.

Suggested Fix

Change the default or routing logic so built-in named evaluators use the supported criteria type (likely "score_model" with appropriate prompt configuration, or a new azure_ai_evaluator-equivalent type once the API exposes it). All three evaluation paths (EvaluateAsync, EvaluateTracesAsync, EvaluateFoundryTargetAsync) call FoundryEvalConverter.BuildTestingCriteria which produces WireTestingCriterion instances — fixing the wire model type fixes all paths.

Environment

Item	Value
Package	Microsoft.Agents.AI.Foundry 1.12.0-preview.260629.1
Azure.AI.Projects	2.1.0-beta.3
Foundry endpoint format	https://<hub>.services.ai.azure.com/api/projects/<project>
.NET	net10.0
Auth	DefaultAzureCredential (working, HTTP 400 is not an auth error)

Workaround

Wrap FoundryEvals in a try/catch and degrade gracefully to local AgentEval metrics. See AgentEval samples 12 & 13 for a reference implementation of this pattern.

[joslat]

@giles17 - @puwocabawo81 - why do you attach a zip file containing an executable there? that is pretty suspicious the least.

[joslat]

@rogerbarreto @moonbox3 doesn't this seems like social engineering / supply-chain attack?

[moonbox3]

@joslat handled.