DesiData-Bench · v1
975 questions evaluated closed-book across 65 Indian government sources. Every answer key is a verbatim quote from the source document.
Open models ranked by SQuAD token-F1 across 975 questions.
| # | Model | F1 | Exact match | Questions evaluated | Raw outputs |
|---|---|---|---|---|---|
| 1 | qwen/qwen3.8-27bgroq | 15.0 | 1.2 | 975 | download JSON |
| 2 | openai/gpt-oss-20bgroq | 13.7 | 1.1 | 975 | download JSON |
| 3 |
| openai/gpt-oss-120bgroq |
| 12.8 |
| 0.9 |
| 975 |
| download JSON |
Closed-book, zero-shot, temperature 0, benchmark v1.0. Scores are SQuAD token-F1 and exact match. Question counts show how many of the benchmark's questions were answered. Every model's raw predictions are downloadable — reproduce them yourself.
Every Saturday afternoon · IST
Read run announcements, suggest models to test, and discuss the benchmark with the community.
Real entries from v1. Answers are verbatim quotes from the source document.
What is the number of male cases for Haryana in measles cases and deaths in India in 2019?
Answer: 62
cbhidghs.mohfw.gov.in · table
What is the percentage of schools having drinking water facility within sch for Bihar in number of schools by availability of drinking water facility within school premises and major source of drinking water – all types of management, 2025-26?
Answer: 99.9
dashboard.udiseplus.gov.in · table
Under which rules is the submission of the environmental statement prescribed?
Answer: the Environment (Protection) Rules, 1986
newmangaloreport.gov.in · factual
What proportion of the global population is estimated to rely on synthetic fertilizers?
Answer: just under half of the people alive today
ourworldindata.org · factual
Designed so nobody has to trust us — only the data and the arithmetic.
Every question was extracted from an official Indian government document — NABARD policies, Finance Commission reports, UGC handbooks, ICAR annual reports, census publications and 60+ other sources — as part of the DesiData catalogue. Each answer is a verbatim span from the source document, with the page and quote recorded in the underlying dataset. Only rows that passed validation were sampled.
Models receive the question alone — no context, no retrieval, no internet. This measures what a model actually learned about India during training, not how well it can search. The exact prompt template is published above; every model gets the identical template.
Answers are scored with the SQuAD exact-match (EM) and token-F1 metrics after standard normalisation (lowercase, strip punctuation/articles). These are the same metrics used by the research community for a decade — nobody has to trust our judgement, only arithmetic.
The benchmark file is versioned and downloadable, with a published SHA-256. Every leaderboard entry links its raw model outputs. Refreshes (v1.1, v2…) will be cut from newly published datasets, so the benchmark grows as the catalogue grows.
Answer the question below with a short, factual answer. If you do not know, give your best guess — do not refuse.
Question: {question}
Answer:Honest limitations: v1 covers English-language extractive QA, so it favours models with strong English recall; long-answer questions are harder under exact match than short-answer ones — which is why we report F1 alongside EM; and questions drawn from documents published after a model's training cutoff may be unanswerable from memory, which is the benchmark working as intended.
Sampled across the catalogue so no single publisher dominates.
Download the JSON, send each question to your model with the prompt template above, and score predictions with SQuAD EM/F1 (any standard implementation). Submit your run — include model name, version, date, temperature and the raw predictions file — and we'll verify and publish it.
If DesiData-Bench contributes to your research, cite the benchmark and the catalogue it was built from.
@misc{desidata-bench-v1,
title = {DesiData-Bench v1: Closed-book QA
over Indian government reports},
author = {DesiData},
year = {2026},
url = {https://www.desidata.in/bench}
}