EDGAR will hand a model a 9 MB filing and then throttle it for asking twice. The interesting part of this server is everything between the model and the wire — and the first version of both the pacer and the cache was wrong in a way that looked fine.
A model does not issue requests in a smooth stream — it fans out several tool calls at once. That is exactly the shape a token bucket lets through. Press the button and watch both approaches against SEC's ceiling of ten requests per second.
A bucket seeded full at capacity 10, refilling at 10/s, averages ten per second — and releases ten instantly, then ten more over the following second. Measured: 19 inside one second, 13.3/s sustained.
Replaced with a strict pacer spacing
grants 1/rate apart, configured at 9/s rather than 10 because
an EDGAR block is sticky. Measured: 9.2/s, peak 10.
The visualisation runs the two algorithms exactly as written,
with no timing jitter, so the bucket hits its theoretical worst case of
capacity + rate = 20 in a second. The Python benchmark firing 40
real concurrent requests measured 19 — scheduling noise shaves one off. Either
way it is roughly double a limit it was written to enforce.
The first cache did the standard thing: store the body with its
validators, revalidate with If-None-Match, treat a 304 as a hit.
It had a hit counter, so it looked like a cache. Every "hit" was a full
re-download.
| host | ETag | Last-Modified | can revalidate? |
|---|---|---|---|
| data.sec.gov | absent | absent | no — always a full 200 |
| www.sec.gov/Archives | absent | present | yes, but unnecessary |
data.sec.gov — submissions, companyfacts,
companyconcept, frames — sends no validators at all, so a conditional
request there is structurally incapable of ever returning 304. Archives
documents are immutable: once an accession is accepted the file
never changes, so a cached copy is correct forever and needs no
revalidation at all.
Freshness had to be decided per host:
immutable for Archives, a one-hour TTL for data.sec.gov because
there is no better option. That is a bounded-staleness tradeoff, not a
correctness guarantee, and the README says so.
Modern filings are inline XBRL. The document opens with a
display:none block holding every tagged fact, so a tag-stripper
that only skips <script> and <style>
hands the model this as the first characters of Apple's FY2025 10-K.
At a default window of 40,000
characters, roughly a quarter of the first window was machine tags. Filing
text is returned windowed with a next_offset cursor — a 10-K
does not belong in a context window whole.