Research Vision

Evaluation Integrity for AI Systems

I develop evaluations whose results both measure the intended capability and remain attributable to that capability—rather than to prior exposure, benchmark-specific optimization, or a proxy that misses the behavior that matters.

Read Research Statement View Publications

An Integrity Framework

A benchmark score becomes credible evidence of capability only when two complementary requirements hold.

Requirement 01

Measurement Validity

Does the evaluation capture the capability it is intended to measure?

  • Observed behavior
  • Execution grounding
  • Evolving worlds
Requirement 02

Attribution Integrity

Does success reflect capability rather than familiarity with or optimization for the evaluation?

  • Private evaluation
  • Exposure diagnosis
  • Repair & maintenance

Research Foundations

My research developed through a progression from verifying software state to asking what AI evaluation results can legitimately establish.

01 · SYSTEM STATE

State Reconciliation

Compare the state a system produces with the state it was intended to produce.

02 · OBSERVED EFFECTS

Execution-Grounded Evaluation

Judge generated solutions by their effects in an environment, not by surface plausibility.

03 · CAPABILITY CLAIMS

Evaluation Integrity

Determine what success measures and whether it can be attributed to the capability under test.

5,110 IaC defects studied 9 previously unknown compiler defects 126 real-world ExITBench tasks 733 execution-based test cases

Maintaining Integrity Across the Benchmark Life Cycle

I study how evaluation integrity can be preserved before exposure, recovered after exposure, and sustained as models and training data change.

01

Preserve

Evaluate black-box models while keeping protected benchmark data private.

02

Diagnose

Identify when exposure has changed behavior enough to overstate capability.

03

Repair

Remove exposure-driven advantages while preserving the capability being tested.

04

Maintain

Adapt evaluations as models evolve and diagnostic value erodes over repeated use.

Evaluation Without Exposure

The Private Evaluation Protocol estimates performance from limited black-box interaction while keeping the private remainder hidden. My current work studies informative interactions, behavioral agreement, and transfer across nearby benchmarks.

Diagnosing and Repairing Exposure

I study modality-specific signatures of exposure, how to distinguish familiarity from generalization, and whether meaning-preserving transformations can recover diagnostic value.

Research Frontier

Evaluating AI in Evolving Worlds

Beyond preserving fixed benchmarks, I envision evaluation as a generative task: constructing controlled worlds with actors, objectives, constraints, and changing conditions that reveal how models reason, interact, decide, and adapt over time.

Capability→Required mechanisms
Generated world→Model interaction
Observed behavior→Capability evidence
MECHANISMS

Which mechanisms of real-world situations must an evaluation preserve?

SUFFICIENCY

What is the smallest generated world sufficient to measure a capability?

TRANSFER

Does success in the generated world provide reliable evidence beyond the simulation?