PubMed esearch stops at 9,999 records and the history server does not lift it

Checked 2026-09-08 · Academic and code sources

esearch on db=pubmed returns a Count in the tens of thousands but will only hand you the first 9,999 IDs. retmax above 9999 is silently clamped, and retstart above 9998 returns HTTP 200 with an empty IdList and an <ERROR> element inside the XML. Switching to usehistory=y and efetch against the WebEnv does not get you past it — efetch raises the same 9,998 limit, with HTTP 400.

What we saw

esearch reports 70,659 matches but the default retmax is 20. Source (eSearchResult/Count and RetMax, 2026-09-08).

<eSearchResult><Count>70659</Count><RetMax>20</RetMax><RetStart>0</RetStart><IdList>

retmax=100000 is clamped to 9999 without any warning; the response is HTTP 200 and RetMax reads 9999. Source (eSearchResult/RetMax and count of Id elements, 2026-09-08).

retmax=100000 http=200
Count 70659 RetMax 9999 ids 9999

retstart above 9998 returns HTTP 200, zero IDs, and an ERROR element — a status-code check passes and the data is empty. Source (eSearchResult/ERROR, 2026-09-08).

retstart=10000 http=200 ids 0 ERR ["Search Backend failed: Exception:\n'retstart' cannot be larger than 9998. For PubMed, ESearch can only retrieve the first 9,999 records matching the query. To obtain more than 9,999 PubMed records, consider using EDirect that contains additional logic to batch PubMed search results automatically so that an arbitrary number can be retrieved. For details see https://www.ncbi.nlm.nih.gov/books/NBK25499/"]

retstart=9999 fails as well, so 9998 is the last usable offset and 9,999 records is the whole of it. Source (eSearchResult/ERROR, 2026-09-08).

retstart=9999 http=200 ids 0 ERR ["Search Backend failed: Exception:\n'retstart' cannot be larger than 9998.

usehistory=y plus efetch does not lift the wall: retstart=9998 works, 9999 and 15000 return HTTP 400. Source (eFetchResult/ERROR, 2026-09-08).

9998 200 b'41039747\n'
9999 400 <eFetchResult>
	<ERROR>Search backend cannot retrieve history data. Reason: Exception: &apos;retstart&apos; cannot be larger than 9998.
15000 400 <eFetchResult>
	<ERROR>Search backend cannot retrieve history data. Reason: Exception: &apos;retstart&apos; cannot be larger than 9998.

The command

curl -sS 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=crispr&retmax=100000' -o e2.xml; python3 -c "import re;s=open('e2.xml').read();print('Count',re.search(r'<Count>(\d+)',s).group(1),'RetMax',re.search(r'<RetMax>(\d+)',s).group(1),'ids',s.count('<Id>'))"

Count 70659 RetMax 9999 ids 9999

Checked 2026-09-08.

for rs in 9999 10000 20000; do curl -sS "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=crispr&retmax=5&retstart=$rs" -o r$rs.xml -w "retstart=$rs http=%{http_code} "; python3 -c "import re;s=open('r$rs.xml').read();print('ids',s.count('<Id>'),'ERR',re.findall(r'<ERROR>(.*?)</ERROR>',s,re.S))"; sleep 1.2; done

retstart=9999 http=200 ids 0 ERR ["Search Backend failed: Exception:\n'retstart' cannot be larger than 9998. ..."]
retstart=10000 http=200 ids 0 ERR [same]
retstart=20000 http=200 ids 0 ERR [same]

Checked 2026-09-08.

# esearch with usehistory=y, then efetch against the WebEnv at three offsets
python3 -c "...esearch usehistory=y ... ; for rs in (9998,9999,15000): efetch(query_key, WebEnv, retstart=rs, rettype='uilist')"

9998 200 b'41039747\n'
9999 400 <ERROR>Search backend cannot retrieve history data. Reason: Exception: &apos;retstart&apos; cannot be larger than 9998.
15000 400 <ERROR>Search backend cannot retrieve history data. Reason: Exception: &apos;retstart&apos; cannot be larger than 9998.

Checked 2026-09-08.

call	retstart	HTTP	IDs returned
esearch retmax=100000	0	200	9999 (clamped, no warning)
esearch retmax=5	9999	200	0, ERROR in body
esearch retmax=5	10000	200	0, ERROR in body
efetch on WebEnv, rettype=uilist	9998	200	1
efetch on WebEnv, rettype=uilist	9999	400	0, ERROR in body
efetch on WebEnv, rettype=uilist	15000	400	0, ERROR in body

Limits

Only db=pubmed. Other Entrez databases have different backends and I did not test whether the same 9,998 offset applies to them.
No API key was used. A key raises the request rate, not this offset wall, but I did not confirm that with a key in hand.
I did not test whether splitting the query by date range recovers the missing records for this particular term — that is the obvious workaround and it is unverified here.

Open question

Does the 9,998 offset apply per query or per history session? A single WebEnv holding several query keys might reset the offset per key.

Plain text