Research

Evaluation, interpretability, and safety

Developing evidence about model capability, robustness, uncertainty, and behavior within accountable systems.

In brief

What evidence is sufficient to trust a system for a particular task?

  • Define the intended task and the consequence of failure before selecting measures.
  • Combine capability, robustness, calibration, and workflow-level evaluation.
  • Use interpretability and human review as evidence sources with their own limitations.

Confidence requires an evidence model

An AI system can appear competent while failing in ways that conventional software tests rarely encounter. Its behavior depends on data, prompts, context, model versions, and interactions with surrounding tools. Evaluation must therefore establish what the system can do, under which conditions, and with what kinds of failure.

NYIC’s research agenda combines task evaluation, robustness, calibration, interpretability, and operational safety. We treat these as connected forms of evidence. A capability score describes observed performance. A robustness study tests its stability. An interpretability investigation explores mechanisms. Operational controls determine what happens when the system is uncertain or wrong.

The central question is how to make deployment decisions with evidence that matches the actual use. A generic benchmark can reveal a useful capability while leaving a critical workflow untested. A carefully designed evaluation should explain both what its result supports and what remains outside its scope.

Begin with task validity

Task validity asks whether a test measures the work that matters. A question-answering benchmark may reward concise correct answers, while an enterprise investigation requires evidence, unresolved issues, and a reviewable recommendation. Optimizing for the benchmark can leave the real workflow unchanged.

We start by specifying decisions, users, inputs, and consequences. A task contract describes what counts as completion and identifies errors that should block release. It also defines acceptable alternatives where several responses are defensible. This is especially important for analytical work in which exact-match scoring can penalize useful answers and reward shallow ones.

HELM provides an influential example of broad language-model evaluation across scenarios and multiple dimensions. Its approach reinforces the value of reporting more than one aggregate measure. NYIC’s perspective extends that principle into workflow-specific scorecards and explicit release decisions. Read the HELM paper.

Build a layered evaluation system

Different checks answer different questions. Unit-level checks can validate schemas, calculations, and state transitions. Component evaluations can measure retrieval or extraction. End-to-end tasks can test the complete workflow. Adversarial scenarios can expose boundary failures that ordinary examples rarely trigger.

The layers should share a failure taxonomy. If a final answer is wrong, reviewers need to distinguish missing evidence, incorrect retrieval, misread evidence, invalid inference, and misleading presentation. This taxonomy turns evaluation into a development instrument. A larger model may help one failure category while leaving another untouched.

A useful evaluation collection includes representative routine cases, difficult cases, historical failures, and deliberately constructed stress tests. These groups should remain distinguishable in reporting. Combining them into a single average can make the result hard to interpret because their prevalence differs from production traffic.

Conceptual evaluation architecture showing deterministic checks, task evaluation, and stress scenarios feeding a release decision and monitored feedback.

Open the conceptual diagram at full size

Conceptual NYIC diagram. Multiple evidence layers inform release, and operational observations become new evaluation cases.

A worked release decision

Imagine a synthetic assistant that drafts responses to internal policy questions. It retrieves documents and cites the passages supporting its answer. Its deployment decision depends on evidence quality and access controls as well as answer usefulness. The following scorecard illustrates a review structure, not an NYIC result.

Dimension Test question Release consequence
Task correctness Does the answer apply the controlling policy? Investigate consequential errors
Evidence support Do cited passages support each material claim? Require traceable support
Freshness Is a superseded policy excluded or identified? Block stale authoritative answers
Access control Can a user receive restricted information? Block unauthorized disclosure
Abstention Does the system recognize missing evidence? Review unsupported certainty
Recovery Does a failed retrieval produce a clear fallback? Require a usable failure state

A single weighted score is a poor fit for some of these dimensions. Strong answer quality cannot compensate for an access-control failure. The release process should separate hard constraints from optimizable objectives. It should also define who can accept residual limitations and what monitoring must accompany deployment.

Robustness under realistic change

Robustness means maintaining appropriate behavior as conditions change. Relevant changes include paraphrased requests, unfamiliar document layouts, new terminology, incomplete evidence, and unexpected tool responses. Some transformations should leave the answer unchanged. Others should change it decisively. Both belong in the test suite.

We use paired scenarios to isolate sensitivity. One pair might differ only in a document’s effective date. Another might change a currency unit. A third might replace a relevant source with a plausible distractor. These tests make it easier to identify whether the system attends to the distinctions that matter.

Adversarial testing adds intentional pressure. Untrusted documents can contain instructions that conflict with the user’s task. Tool responses can include misleading metadata. An attacker can attempt to redirect retrieval or induce an unauthorized action. Evaluation should exercise these paths while the application enforces authority through mechanisms outside the model’s generated text.

Calibration and selective assistance

Confidence should correspond to observed reliability if it is used to determine action. A model saying it is certain provides little evidence on its own. Calibration requires a defined prediction event, a confidence measure, and observed outcomes across comparable cases.

Research on neural-network calibration examines the relationship between predicted confidence and empirical correctness. The methods and findings depend on the model and task. Applying them to generative workflows requires care because a response can contain several claims with different levels of support. Read the calibration paper.

For enterprise systems, selective assistance can be more useful than a universal confidence number. The system can answer when sufficient evidence is available, request clarification when the task is ambiguous, and route consequential uncertainty to review. Evaluation should measure both the quality of answered cases and the burden created by abstention.

Thresholds require ongoing validation. A threshold selected on a balanced test set may behave differently when production inputs shift. Monitoring should inspect the kinds of cases accepted, deferred, and corrected. Changes in that distribution can matter before a headline accuracy measure moves.

Interpretability and its limits

Interpretability research investigates relationships between internal computation and observable behavior. It can help generate hypotheses about representations, identify features associated with behavior, and support controlled interventions. It is a complementary source of evidence rather than a universal certification of reliability.

Sparse-autoencoder research explores decomposing model activations into features that can be more interpretable than individual neurons. This is an important direction for understanding distributed representations. A feature description still needs validation, and a local interpretation does not establish complete understanding of the model. Read the sparse-autoencoder study.

We distinguish mechanistic evidence from user-facing explanation. A generated explanation can help a person understand an answer without faithfully describing the model’s internal process. Evidence citations, reproducible calculations, and explicit assumptions remain valuable even when a complete mechanistic account is unavailable.

Human oversight as a designed interaction

Oversight works only when a reviewer has the information, time, and authority to intervene. A generic approval button can create the appearance of control while encouraging automatic acceptance. Review interfaces should surface the proposed action, supporting evidence, uncertainties, and consequences at the moment of decision.

Evaluation should include the human workflow. Measure whether reviewers detect important errors, how long review takes, and whether the system’s presentation biases decisions. Excessive alerts can reduce attention. Missing context can make rejection difficult. These are design issues that belong in the safety case for the deployed application.

Traceability connects oversight with improvement. A useful record identifies model and prompt versions, relevant evidence, tool actions, review decisions, and the final outcome. Logging should collect only what is justified for operation and debugging, with appropriate access and retention controls.

Turning evidence into release discipline

A release decision should record the tested configuration, evaluation scope, unresolved failures, and operating restrictions. Changing the model, retrieval pipeline, tool permissions, or document corpus can invalidate part of that evidence. Regression evaluation should follow the affected behavior rather than relying on a fixed calendar alone.

NYIC’s research connects evaluation to this practical discipline. We aim to develop clearer tests, better uncertainty signals, and more informative accounts of failure. The resulting systems should make their operating boundaries observable and give institutions a sound basis for expanding capability as evidence improves.

Sources & further reading

  1. Holistic Evaluation of Language Models ↗
  2. On Calibration of Modern Neural Networks ↗
  3. Sparse Autoencoders Find Highly Interpretable Features in Language Models ↗