download.bls.gov 403: the User-Agent must contain an email address

Checked 2026-09-08 · Statistical agencies: BLS, Census, BEA, FRED and EIA

download.bls.gov returns 403 Access Denied to every request whose User-Agent does not contain something shaped like an email address. Put any string matching name@domain.tld in the User-Agent and the same URL returns 200. A browser User-Agent does not work; neither does an empty one, nor curl's default, nor "contact AT example DOT com".

What we saw

A plausible browser User-Agent is refused with 403 and a 1321-byte HTML page. Source (HTTP status, 2026-09-08).

== UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36
HTTP 403 bytes=1321

The same URL with an email-shaped User-Agent returns the 15 MB tab-separated series catalogue. Source (HTTP status and first line, 2026-09-08).

== UA: opendatanotes.org (notes@opendatanotes.org)
HTTP 200 bytes=15288538
series_id        	lfst_code	periodicity_code	series_title	absn_code	activity_code	ages_code	cert_code	class_code	duration_code	education_code	entr_cod

An at-sign alone is not enough. "x@y" is refused; "x@y.co" is served. The gate wants a dotted domain after the at-sign. Source (HTTP status per User-Agent, 2026-09-08).

== UA: [x@y] HTTP 403 bytes=1321
== UA: [@] HTTP 403 bytes=1321
== UA: [abc] HTTP 403 bytes=1325
== UA: [] HTTP 403 bytes=1325

The address can be anywhere in the string and the domain is not validated; a two-letter and an invented TLD both pass. Source (HTTP status per User-Agent, 2026-09-08).

== UA: [x@y.com] HTTP 200
== UA: [x@y.co] HTTP 200
== UA: [a@b.zz] HTTP 200
== UA: [hello x@y.com world] HTTP 200
== UA: [x@y.comm] HTTP 200

The 403 body says why, but does not say what to change. It gives a per-request error code and a contact form. Source (403 body text, 2026-09-08).

<h2>Access Denied</h2>
<p>The BLS is committed to providing data promptly and according to established schedules. Automated retrieval programs (commonly called "robots" or "bots") can cause delays and interfere with other customers' timely access to information. Therefore, bot activity that doesn't conform to BLS usage policy is prohibited.</p>

Directory listings are served over the same gate, so the flat-file tree can be walked once the User-Agent passes. Source (directory index, 2026-09-08).

<html><head><title>download.bls.gov - /pub/time.series/ln/</title></head><body><H1>download.bls.gov - /pub/time.series/ln/</H1><hr>

<pre><A HREF="/pub/time.series/">[To Parent Directory]</A><br><br>  9/4/2026  8:30 AM          104 <A HREF="/pub/time.series/ln/ln.absn">ln.absn</A>

The command

for ua in "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" "curl/8.0" "opendatanotes.org (notes@opendatanotes.org)"; do curl -sS -o /tmp/f.out -w "HTTP %{http_code} bytes=%{size_download}\n" -A "$ua" 'https://download.bls.gov/pub/time.series/ln/ln.series'; done

HTTP 403 bytes=1321
HTTP 403 bytes=1325
HTTP 200 bytes=15288538

Checked 2026-09-08.

for ua in "x@y" "@" "abc" "" "Mozilla/5.0 (contact: notes@opendatanotes.org)" "notes AT opendatanotes DOT org"; do curl -sS -o /dev/null -w "HTTP %{http_code}\n" -A "$ua" 'https://download.bls.gov/pub/time.series/ln/ln.txt'; done

== UA: [x@y] HTTP 403 bytes=1321
== UA: [@] HTTP 403 bytes=1321
== UA: [abc] HTTP 403 bytes=1325
== UA: [] HTTP 403 bytes=1325
== UA: [Mozilla/5.0 (contact: notes@opendatanotes.org)] HTTP 200 bytes=14808
== UA: [notes AT opendatanotes DOT org] HTTP 403 bytes=1325

Checked 2026-09-08.

curl -sS -A 'x@y.com' 'https://download.bls.gov/pub/time.series/ln/' | head -c 400

<html><head><title>download.bls.gov - /pub/time.series/ln/</title></head><body><H1>download.bls.gov - /pub/time.series/ln/</H1><hr>

<pre><A HREF="/pub/time.series/">[To Parent Directory]</A><br><br>  9/4/2026  8:30 AM          104 <A HREF="/pub/time.series/ln/ln.absn">ln.absn</A>

Checked 2026-09-08.

User-Agent sent	HTTP	body size
(none sent)	403	1325
curl/8.0	403	1325
abc	403	1325
Mozilla/5.0 (X11; Linux x86_64) ... Chrome/126.0 Safari/537.36	403	1321
@	403	1321
x@y	403	1321
notes AT opendatanotes DOT org	403	1325
x@y.co	200	-
a@b.zz	200	-
x@y.comm	200	-
hello x@y.com world	200	-
Mozilla/5.0 (contact: notes@opendatanotes.org)	200	14808

Limits

The rule was inferred from twelve User-Agent strings, not read from a published filter. It is consistent with everything tested here, but a different pattern could be the real trigger.
This is a separate host from api.bls.gov and appears to be separately gated: the flat files were still served after the API had refused this address for exceeding its daily cap. The two limits were not tested against each other deliberately, so treat the independence as an observation, not a guarantee.
No download rate was tested. The 403 page is about bot activity generally, so an email-shaped User-Agent almost certainly does not exempt a fast crawler.
Whether the gate is on the origin or an edge cache was not determined; the 403 page prints an akamai-prefixed contact link, which suggests the edge.

Open question

The two refusal bodies differ by four bytes (1321 vs 1325) depending on the User-Agent sent. Two different rules, or just the error code string echoing something?

Plain text