Issues · EleutherAI/lm-evaluation-harness · GitHub

Streamlining lm-eval Architecture
#3083 · baberabb opened on Jun 23, 2025
1
Datasets with loading scripts - no longer supported
#3171 · baberabb opened on Jul 21, 2025
15

Labels Milestones

[Task Proposal] Add JailbreakBench (JBB-Behaviors) — adversarial-prompt robustness eval

#3915

· neerazz opened

on Jul 6, 2026

Recent main merges introduce pre-existing lint violations that block PR CI

#3900

· bongho opened

on Jul 1, 2026

InfiniteBench tasks fail on a clean install

#3895

· gbyu-amd opened

on Jun 29, 2026

Request cache key ignores generation_kwargs -> silently reuses cached instances across different sampling parameters

#3881

· AUTHENSOR opened

on Jun 24, 2026

Version string uses .dev0 suffix which conflicts with PEP 440 release tagging

#3877

· kpunwatk opened

on Jun 24, 2026

[Proposal] New Task: Turkish Clinical Source Support (clinical_safety group)

#3866

· goktugozkanmd opened

on Jun 21, 2026

SCROLLS tasks fail to load with recent datasets (dataset scripts no longer supported)

#3859

· bongho opened

on Jun 16, 2026

[Task Proposal] Add SAS-Bench — Fine-Grained Short Answer Scoring (arXiv:2505.07247)

#3849

· Man-ishu678 opened

on Jun 15, 2026

Unable to load XNLI, XCOPA, PAWS-X, and XQuAD without specifying namespace

#3846

· mrqorib opened

on Jun 14, 2026

Paired significance test for comparing two runs (model_comparator.py applies an unpaired z-test to paired data)

#3831

· yongzhe2160cs opened

on Jun 11, 2026

Guidance on contributing refute_soundness task (scientific critique benchmark)

#3820

· connerlambden opened

on Jun 5, 2026

Evaluation metrics are lower with all2all-backend=pplx than with naive all2all manager under EP-enabled MoE inference

#3782

· NorthWindAqa opened

on May 23, 2026