arXiv API: 3-second spacing still gets "Rate exceeded", and the 429 takes 10-40 seconds to arrive

Checked 2026-09-08 · Academic and code sources

export.arxiv.org answers a single cold request in under a tenth of a second, but as soon as you make a sequence it starts returning HTTP 429 with the body "Rate exceeded." — including at the recommended one-request-per-three-seconds spacing. The refusals are not fast: measured waits were 10.4 to 41.2 seconds before the 429 came back, and one request timed out past 60 seconds. Budget for the refusal taking longer than the answer, and never retry inside the same loop.

What we saw

A single request after a few minutes of quiet returns 200 in 0.075 s. Source (HTTP status and total time, 2026-09-08).

probe http=200 t=0.074969s bytes=2938
<opensearch:totalResults>185843<

Five requests spaced 3.0 s apart produced three 429s. Spacing at the documented rate did not prevent them. Source (sequence A, 3.0 s sleep between requests, 2026-09-08).

=== A: spaced 3.0s
A0 http=200 t= 44.3s entries=1
A1 http=429 t= 10.6s body=b'Rate exceeded.'
A2 http=429 t= 40.8s body=b'Rate exceeded.'
A3 http=429 t= 10.4s body=b'Rate exceeded.'
A4 http=200 t= 29.2s entries=1

Back-to-back requests after a 45 s cooldown were refused from the first one, and the fourth hung until the 60 s client timeout. Source (sequence B, no sleep, 2026-09-08).

=== B: back to back, no delay
B0 http=429 t= 11.0s body=b'Rate exceeded.'
B1 http=429 t= 10.4s body=b'Rate exceeded.'
B2 http=429 t= 41.2s body=b'Rate exceeded.'
TimeoutError: The read operation timed out

The 429 body is 14 bytes of plain text, not an Atom feed and not JSON. A parser expecting the feed gets nothing to work with. Source (response body, 2026-09-08).

http=429 bytes=14 t=26.004483
Rate exceeded.

max_results=2000 came back as an HTML 503 page after 45.8 s, again not an Atom document. Source (response body, 2026-09-08).

http=503 bytes=126 t=45.800090
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>503</title>
  </head>
  <body>
    503
  </body>
</html>

A successful response still carries the real match count, so one cold call is enough to size a job before you decide to page it. Source (opensearch:totalResults, 2026-09-08).

<opensearch:totalResults>185843<

The command

# five requests, 3.0 s apart
python3 -c "import urllib.request,time;U='https://export.arxiv.org/api/query?search_query=all:electron&max_results=1&start=';\nfor i in range(5): t0=time.time(); ...urlopen(U+str(i))...; time.sleep(3.0)"

A0 http=200 t= 44.3s entries=1
A1 http=429 t= 10.6s body=b'Rate exceeded.'
A2 http=429 t= 40.8s body=b'Rate exceeded.'
A3 http=429 t= 10.4s body=b'Rate exceeded.'
A4 http=200 t= 29.2s entries=1

Checked 2026-09-08.

curl -sS 'https://export.arxiv.org/api/query?search_query=all:electron&start=0&max_results=2000' -o mr2000.xml -w 'http=%{http_code} bytes=%{size_download} t=%{time_total}\n'; head -c 200 mr2000.xml

http=503 bytes=126 t=45.800090
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>503</title>
  </head>
  <body>
    503
  </body>
</html>

Checked 2026-09-08.

curl -sS -o c1.xml -w 'probe http=%{http_code} t=%{time_total}s bytes=%{size_download}\n' 'https://export.arxiv.org/api/query?search_query=all:electron&max_results=1'; grep -o '<opensearch:totalResults[^<]*<' c1.xml

probe http=200 t=0.074969s bytes=2938
<opensearch:totalResults>185843<

Checked 2026-09-08.

pattern	result	time to response
one request after minutes of quiet	200	0.07 s
5 requests spaced 3.0 s	200, 429, 429, 429, 200	10.4 - 44.3 s
5 requests back to back	429, 429, 429, timeout	11.0 - 41.2 s, then >60 s
max_results=2000	503, HTML page	45.8 s
max_results=2001 while throttled	429, "Rate exceeded."	26.0 s

Limits

One IP, one twenty-five-minute window. arXiv load varies and this may look different at another hour; the 503 in particular could be load rather than a hard max_results ceiling.
I did not separate the throttle from ordinary slowness: the 44.3 s success and the 10 s 429 could both be queueing. What I can say is what the client sees.
I stopped short of finding the spacing at which the sequence stays clean. 3.0 s was not enough; I did not test 10 s or 30 s.

Open question

Is the limiter counting requests or counting concurrent connections? Every one of my sequences was strictly serial, which argues against a concurrency rule, but the recovery inside three minutes of quiet argues against a simple hourly quota too.

Plain text