--- name: federal-register-scraper description: Read US Federal Register documents as structured records via the Apify Actor arman-bd/federal-register-scraper. Returns the document number, title, type, agency abstract, issuing agencies and their slugs, publication date, effective date, comment deadline, docket IDs, regulation ID numbers, the significance flag and links to the official page and PDF. Use for regulatory monitoring, comment-deadline diaries, executive-order tracking, agency rulemaking feeds and RegTech backfills. Not for full document text, state or foreign regulators, comment submissions, or documents that have not yet been published. --- # Federal Register Scraper: Rules, Notices & Executive Orders Apify Actor `arman-bd/federal-register-scraper`. Filter by text, agency, document type and publication date, and get one dataset record per Federal Register document. It runs without credentials. ## When to use it - A daily or weekly compliance feed for one agency's rulemaking. - Building a diary of `commentsCloseOn` deadlines on proposed rules in your sector. - Tracking executive orders and other presidential documents over a period. - Backfilling a topic across a date range, using synonym terms that are unioned for you. - Filtering the whole Register down to the economically significant subset. ## When not to use it - Full document text. You get the agency's own abstract plus links to the official page and PDF; the body itself is often hundreds of pages and is not fetched. - State registers, foreign regulators, the Code of Federal Regulations, or the Unified Agenda. Federal Register documents only. - Submitting or reading public comments. Only the deadline field is here. - Anything before publication. A rule appears the day it is published, not when signed. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/federal-register-scraper').call({ searchTerms: ['pfas'], agencies: ['environmental-protection-agency'], documentTypes: ['RULE', 'PRORULE'], fromDate: '2026-01-01', maxResults: 1000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~federal-register-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"agencies":["environmental-protection-agency"],"documentTypes":["RULE","PRORULE"],"fromDate":"2026-01-01","maxResults":200}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/federal-register-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchTerms` | string[] | no | `[]` | Full-text terms over title, abstract and body. Each term runs as its own query and the results are unioned and de-duplicated on `documentNumber`, so synonyms cost extra requests but never extra rows. Empty means no text search at all. | | `agencies` | string[] | no | `[]` | Agency slugs such as `environmental-protection-agency`. Display names and agency page URLs are accepted and slugified. Multiple agencies are OR-ed. Empty means every agency. | | `documentTypes` | string[] | no | `[]` | Any of `RULE` (final rule), `PRORULE` (proposed rule), `NOTICE`, `PRESDOCU` (presidential document). Common long forms such as `Final Rule` and `Executive Order` are mapped to the right code. Empty means all four. | | `fromDate` | string | no | `""` | Publication date lower bound, strictly `YYYY-MM-DD`. Anything else throws before a request is made. | | `toDate` | string | no | `""` | Publication date upper bound, same format. Empty runs through to today's issue. A `fromDate` later than `toDate` throws. | | `significantOnly` | boolean | no | `false` | Keep only documents flagged significant under Executive Order 12866. | | `maxResults` | integer | no | `1000` | Total documents saved across every query combined. `0` means no cap beyond what can be paged. | **The date window is the only lever that gets you past the ceiling.** Every filter here is applied at the source before anything is downloaded, and filters combine with AND while multiple values inside one filter are OR-ed, so `agencies` of two with `documentTypes: ["RULE"]` means final rules from either agency. `searchTerms` is the exception, being separate queries unioned afterwards. The constraint that actually shapes a run is that one query cannot page past roughly 10,000 documents, whatever the real match count. Backfilling history means walking `fromDate` and `toDate` a month or a quarter at a time, not asking for everything at once. ## Output One record per document saved. | Field | Type | Notes | |---|---|---| | `documentNumber` | string | Federal Register document number, for example `2026-16083`. The stable identifier and the de-duplication key. | | `title` | string \| null | Full document title. Often very long. | | `type` | string \| null | Display form, not the input code: `Rule`, `Proposed Rule`, `Notice`, `Presidential Document`. | | `abstract` | string \| null | The agency's own summary. `null` for types that carry none, notably many presidential documents. | | `agencies` | string[] | Issuing agency display names. | | `agencySlugs` | string[] | The same agencies as slugs. Feed these straight back into the `agencies` input. | | `publicationDate` | string \| null | `YYYY-MM-DD`. This is the date the clock starts on. | | `effectiveOn` | string \| null | Date the rule takes effect, where the document sets one. Usually `null` on notices and proposed rules. | | `commentsCloseOn` | string \| null | Comment deadline. The field that matters on proposed rules. | | `docketIds` | string[] | Agency docket identifiers, for example `EPA-R09-OAR-2026-3797`. `[]` when none. | | `regulationIdNumbers` | string[] | RINs linking the document to the Unified Agenda. `[]` when none. | | `htmlUrl` | string \| null | Canonical page for the document. | | `pdfUrl` | string \| null | Official PDF of the issue page range. | | `significant` | boolean \| null | `true` when flagged significant under EO 12866. `null` means unknown, not "not significant". | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, an EPA proposed rule with long strings trimmed: ```json { "documentNumber": "2026-16083", "title": "Partial Approval and Partial Disapproval of Air Quality State Implementation Plans; Arizona; Prevention of Significant Deterioration Infrastructure Requirements …", "type": "Proposed Rule", "abstract": "The Environmental Protection Agency (EPA) is proposing to partially approve and partially disapprove a revision to the Arizona State implementation plan (SIP) …", "agencies": ["Environmental Protection Agency"], "agencySlugs": ["environmental-protection-agency"], "publicationDate": "2026-08-06", "effectiveOn": "2026-09-08", "commentsCloseOn": "2024-06-13", "docketIds": ["EPA-R09-OAR-2026-3797", "FRL-13389-01-R9"], "regulationIdNumbers": [], "htmlUrl": "https://www.federalregister.gov/documents/2026/08/06/2026-16083/partial-approval-and-partial-disapproval-of-air-quality-state-implementation-plans-arizona", "pdfUrl": "https://www.govinfo.gov/content/pkg/FR-2026-08-06/pdf/2026-16083.pdf", "significant": null, "scrapedAt": "2026-08-06T11:42:18.559Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "queriesRequested": 2, "queriesFailed": 0, "failures": [], "documentsSaved": 119, "duplicatesSkipped": 4, "perQuery": [ { "term": "pfas", "reported": 119, "fetched": 119, "saved": 115, "truncated": null } ], "filters": { "searchTerms": ["pfas"], "agencies": ["environmental-protection-agency"], "documentTypes": ["RULE", "PRORULE"], "fromDate": "2026-01-01", "toDate": "", "significantOnly": false, "maxResults": 1000 }, "finishedAt": "2026-08-06T11:42:18.559Z" } ``` `perQuery[].truncated` is the field to branch on. `null` means that query ran to completion. A non-null string says why it stopped: the cap was reached, the query hit the 10,000-document ceiling and its true total is unknown, or fewer documents were reachable than `reported`. In the ceiling case, split the date range and re-run. `failures` entries carry `query` and `error`; a run where every query failed throws instead of finishing. `saved` below `fetched` within a query is de-duplication against earlier queries, not loss. ## Behaviour to plan around - **`type` in the output is not the code you sent in.** You filter with `PRORULE` and you read back `Proposed Rule`. Map the four display strings yourself when grouping. - **The 10,000 ceiling is invisible in the counts.** The reported match total saturates at exactly 10,000, so a query matching 50,000 documents also reports 10,000 and fetching all of them looks complete. Reaching the ceiling is itself the only signal, and it is reported through `truncated`. Never treat `reported` as a true match count once it sits at 10,000. - **`commentsCloseOn` can be in the past.** Some documents reuse a deadline from an earlier related notice. The value is passed through exactly as published, so validate against `publicationDate` before diarying it. - **`significant: null` is unknown, not false.** The flag depends on a Unified Agenda linkage that many documents do not have. Use `significantOnly` if you want only the confirmed ones. - **Abstracts keep inline source markup.** Tags such as `2.5` mark subscripts in chemical and standards notation and are left intact, because stripping them naively would corrupt the meaning. Strip them at render time if you must. - **A bad filter value fails loudly.** An unknown agency slug comes back as a rejected query with the offending field named, and that lands in `failures` rather than being silently ignored. An exploratory run and the `agencySlugs` of any returned document are the reliable way to learn the exact strings. - **Zero matches is not an error.** A query matching nothing returns a count of zero with no results block at all, and the run continues. - **Transient failures are retried** three times with linear backoff. Rejected filters and not-found responses are final and are not retried. - **`maxResults` is consumed in query order.** With several search terms and a tight cap, later terms may never run. Their absence from `perQuery` is the tell. ## Recipes **Daily compliance sweep for one agency.** Schedule it with `fromDate` set to yesterday. ```json { "agencies": ["environmental-protection-agency"], "documentTypes": ["RULE", "PRORULE"], "fromDate": "2026-08-01", "maxResults": 500 } ``` Alert on any `documentNumber` not seen in the previous run. **Comment-deadline diary on a topic.** Proposed rules only, synonyms unioned so a term choice does not lose documents. ```json { "searchTerms": ["pfas", "per- and polyfluoroalkyl"], "documentTypes": ["PRORULE"], "fromDate": "2026-01-01", "maxResults": 1000 } ``` Keep records where `commentsCloseOn` is non-null and later than today, then sort ascending. **Executive orders for a quarter.** ```json { "documentTypes": ["PRESDOCU"], "fromDate": "2026-04-01", "toDate": "2026-06-30", "maxResults": 0 } ``` Expect `abstract` to be `null` on most of these; the title and `htmlUrl` carry the content. **Backfill a year without hitting the ceiling.** Run this four times, moving the window by a quarter each time, and concatenate the datasets. ```json { "documentTypes": ["RULE", "PRORULE"], "significantOnly": true, "fromDate": "2026-01-01", "toDate": "2026-03-31", "maxResults": 0 } ``` Check `perQuery[].truncated` after each window; if it is non-null, halve the window.