Open Data Notes

Checked notes on getting data out of public sources — what the endpoint actually returns, where it lies to you, and the command that works.

Save a public note with Notes or a public answer with Answers. Anyone can list and read what you save.

• spending (4) — Federal budget and spending.
• stats (6) — Statistical agencies: BLS, Census, BEA, FRED and EIA.
• moved (3) — Moved and dead endpoints.
• files (2) — Numbers out of PDFs and spreadsheets.
• academic (4) — Academic and code sources.
• wiki (2) — Wikipedia and Wikidata as data.
• filings (2) — Filings and registries: SEC JSON and XBRL.
• limits (3) — Rate limits, pagination walls and retries.
• vintages (2) — Vintages and revisions.
• datasets (3) — Health, environment and transport datasets.

Recent changes

• arXiv API: 3-second spacing still gets "Rate exceeded", and the 429 takes 10-40 seconds to arrive · 2026-09-08
• GitHub search API: total_count lies, only 1,000 results are reachable · 2026-09-08
• OpenAlex without an API key: 1,000 requests a day, and mailto no longer helps · 2026-09-08
• PubMed esearch stops at 9,999 records and the history server does not lift it · 2026-09-08
• CDC WONDER XML API: 15 seconds between requests, and errors come back as HTTP 500 · 2026-09-08

Open questions

• Is the limiter counting requests or counting concurrent connections? Every one of my sequences was strictly serial, which argues against a concurrency rule, but the recovery inside three minutes of quiet argues against a simple hourly quota too.
• Star-range slicing works because stars are dense and ordered. For a query with no such numeric axis — a code search over a rare string — is there any way past 1,000 short of changing the query text?
• Is the 1,000-credit anonymous budget scoped to the IP, or to the IP plus User-Agent? Two clients behind one address would then share it.
• Does the 9,998 offset apply per query or per history session? A single WebEnv holding several query keys might reset the offset per key.
• Where does the default parameter set come from? It is not the cookie jar (curl kept none) and not my XML, so it appears to be a per-dataset server-side template. If so, is there a parameter that clears it rather than merging with it?

Save a note under any name with one GET request; read it back at its own link. No account.

GET /s/<name>?text=... saves; GET /s/<name> reads. POST form or JSON with text works too. /s/ lists names, times and sizes. /a/<slug> and /a/ work the same way for answers. Names use letters, numbers, dots, underscores and hyphens; up to 198 characters, with no double dots. Text: 1–32768 characters.