Research Vision

A benchmark score is valuable only when success reflects the capability the benchmark was designed to test. I call this problem evaluation integrity: whether success can be attributed to the capability under test rather than to prior exposure, benchmark-specific optimization, or a proxy that fails to capture the behavior that matters.

My research addresses evaluation integrity across the full life cycle of a benchmark. For private benchmarks, I study how to evaluate black-box models without revealing the benchmark. For exposed benchmarks, I study how to diagnose whether exposure has distorted performance and how to repair the evaluation. Across both settings, I ask how integrity can be maintained as models and training data evolve.

Evaluation Integrity Software Engineering Benchmark Contamination Observed Behavior

Research Program

1
Grounding Evaluation in Observed Behavior
Active

My work on evaluation began with state reconciliation in Infrastructure as Code: correctness should be established from the state a system actually produces, not from the plausibility of the artifact intended to produce it. An empirical study of 5,110 defects led to an eight-category taxonomy and heuristics that uncovered nine previously unknown Ansible compiler defects. I carried that principle into ExITBench, an execution-based benchmark with 126 real-world IT-automation tasks and 733 test cases. Even after ten attempts, the strongest model succeeded on only 23.9% of tasks, with failures concentrated in state-reconciliation reasoning and specialized automation knowledge.

Papers
2
Evaluation Without Exposure
Active

When a benchmark remains private, evaluation itself can create the exposure that later undermines its score. I developed the Private Evaluation Protocol (PEP), which estimates performance on a private benchmark from limited interaction with a black-box model while keeping the private remainder hidden from the provider. On structured benchmarks, PEP estimated accuracy within 5.88 percentage points on average. My ongoing work studies which public or protected interactions are most informative, how to measure behavioral agreement for generation and tool-use tasks, and when evidence transfers across nearby benchmarks.

Papers
  • AACL-IJCNLP 2026 Private Evaluation Protocol: Contamination-Resistant Black-Box Evaluation via Public-Slice Surrogates — Md. Mahadi Hasan Sibat, Sazzadur Rahaman, Yusuf Sarwar Uddin, Santu Karmaker
3
Diagnosing, Repairing, and Maintaining Benchmarks
Active

For an exposed benchmark, the question is not merely whether a model saw the evaluation, but whether exposure changed its behavior enough to overstate capability. I study how exposure signatures differ across tasks and modalities, how to distinguish familiarity from ordinary generalization, and how meaning-preserving transformations can recover diagnostic value. Over repeated use, both private and repaired benchmarks face an additional maintenance problem. I use state reconciliation to frame this challenge: characterize a benchmark's current integrity state, compare it with the state required for reliable evaluation, and intervene only when that gap becomes meaningful.

Research Directions
  • Diagnosis Identify evidence of exposure-driven advantage across benchmarks and modalities.
  • Repair Test whether meaning-preserving transformations remove exposure advantages without changing the capability under test.
  • Maintenance Preserve evaluation integrity as models, training data, and sources of exposure change over time.