Research Direction: Validating Designed Difficulty Against Empirical Model Vulnerability

#453
by jiwon9703 - opened

Hi everyone!

After finishing the difficulty analysis in PR #449, I've been thinking about what we could explore next.

One thing that stood out to me was that evidence scarcity was associated with model divergence, but the relationship wasn't particularly robust across different statistical specifications. The existing difficulty annotations also didn't show a clear improvement in predicting divergence on held-out claims.

That made me wonder whether the difficulty we assign when designing a question is necessarily the same as how difficult that question turns out to be for a model.

I started looking for related work and came across a few papers that seemed relevant. I haven't read them in detail yet, so these are mostly first impressions based on their abstracts.

Related papers

1. Easy2Hard-Bench (NeurIPS 2024)
https://papers.nips.cc/paper_files/paper/2024/hash/4e6f22305275966513990f53cec908e0-Abstract-Datasets_and_Benchmarks_Track.html

This one caught my attention because it estimates question difficulty from observed performance, including approaches based on IRT and Glicko-2. It seems like a useful starting point for thinking about how our designed difficulty labels compare with actual model behavior.

2. Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory (AAAI 2026)
https://ojs.aaai.org/index.php/AAAI/article/view/40814

This paper looks at benchmark measurement quality using IRT. I'm curious whether a similar perspective could help us understand which ConspiracyBench questions actually distinguish between models, rather than just looking at overall divergence rates.

3. Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory (ICML 2026)
https://proceedings.mlr.press/v306/choi26n.html

This also seems relevant since our divergence labels depend on the judge. Before treating observed divergence as a measure of difficulty, it would be useful to understand how much of the variation might come from the evaluation process itself.

4. ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety (TACL 2026)
https://aclanthology.org/2026.tacl-1.96/

This isn't directly about difficulty estimation, but it looks at characteristics of conspiracy-related content and how models respond to it. I wonder whether some of these characteristics could help explain differences in model vulnerability beyond evidence scarcity and question openness.

5. SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (EMNLP 2025)
https://aclanthology.org/2025.emnlp-main.1274/

This is from an educational assessment setting, so it's quite different from ConspiracyBench. Still, the idea of estimating difficulty for unseen questions before collecting actual responses seemed interesting.

Where we could go from here

I think there are a few questions worth looking into.

For example, are questions that cause divergence in one model also challenging for other models? And how much of the observed variation is due to the question itself, rather than the model, repeated sampling, or the judge?

It might also be interesting to estimate empirical vulnerability at the claim level and compare it with our existing difficulty annotations. We could start with simple statistical approaches or hierarchical models before considering something more involved like IRT.

I'm not suggesting that we replace the current difficulty labels. They describe properties of the questions, whereas empirical vulnerability depends on the model and evaluation setup. The relationship between the two is what I'd like to understand better.

Anyway, these are just some initial thoughts. I still need to go through the papers properly, and I'm sure there are relevant studies or approaches I've missed.

If anyone has recommendations for other papers, different ways of approaching this, or even completely different research questions related to difficulty, I'd love to hear them!

Would be great to get some thoughts before deciding what to investigate next.

Interesting Jiwon, when I audited 77 always-diverge cases I made some observations that would relate to what you are writing:

  • designed vs. empirical difficulty: 27 of the 77 cases are on items labelled easy, only 16 on hard
  • mostly model specific: those 77 cases span 50 questions, and 32 of those fail every time for only a single model
  • maybe not one dimension: some failures are models dneying true events that sound like conspiracies, others accepting false premises. These may pull in opposite directions in an IRT model.

Thanks, this is really interesting!

The model-specific failures are particularly interesting, and I hadn't thought much about the possibility that divergence could reflect different, potentially opposing failure modes. That seems important to consider before trying to model empirical difficulty as a single dimension.

I'm also curious about the easy vs. hard cases. Since these are selected from the always-diverge cases, I wonder how the pattern would look after accounting for the total number of items in each difficulty band.

Would you be able to share the question IDs or your audit notes? I'd love to take a closer look and see how these observations relate to the results from PR #449.

This might actually be a good place to start before exploring IRT. Thanks for sharing!

hey guys, I looked into Jiwon's easy vs hard question using corpus.jsonl and PMoeller's cases_audited.csv.

(Counts)
Out of 127 easy items, 18 had a model fail all 5 tries. Medium was 24 of 111 and hard was 8 of 16. So basically hard fails more often, easy just has the most cases because there are way more easy items.

(Suggestion)
Please try using the same items for every model.
I found in #440 Qwen drops 8 items, Nemotron 24 and K2 11, so these numbers may be a bit off. Judge noise matters too, in #454 a model-reader check disagreed with it on 21.5% of 200 replies, but that was a different set so only a rough flag.

(Limits)
Only 16 hard items so 8/16 is rough, and it's only the 5/5 cases from PMoeller's csv.

Thanks for looking into this! The difficulty-band comparison is really helpful, especially since the raw counts gave a somewhat different impression.

I agree that we should use the same set of questions across models and account for missing evaluations and judge uncertainty.

I think I'll take a closer look at the baseline data and try a matched-item analysis, comparing overall divergence rates with consistent 5/5 failures across difficulty levels and models. I'd also like to see whether the patterns differ by failure type, although that may require additional annotations.

I'll share what I find here once I have some results. Thanks again to both of you for the insights!

Thanks @PMoeller and @vnnviv ! I ran the matched-item analysis on the 235 questions where all five models had five eligible runs. At least one model diverged in all five conversations on 12/117 easy questions, 21/102 medium questions, and 8/16 hard questions, counting a conversation if the Judge flagged any turn. Matching removed 10 easy and 9 medium questions. All 16 hard questions stayed in.

On the model-specificity question, 24 of those 41 questions reached 5/5 for just one model. The other 17 reached it for two or more. I’ve put the code, the opening-turn comparison, and the sensitivity checks in PR #471.

stationhouse changed discussion status to closed

Sign up or log in to comment