--- name: wayback-machine-scraper description: Enumerate Internet Archive captures for a domain or URL prefix via the Apify Actor arman-bd/wayback-machine-scraper. Returns one row per capture with the original URL as crawled, the raw Wayback timestamp and its ISO form, a permanent replay link, the archived HTTP status, MIME type, SHA-1 content digest, capture byte length and the canonical sort key. Use when a task needs historical URL discovery for an SEO migration audit, recovering pages that no longer exist, attack-surface research over forgotten hostnames and paths, content-change detection by digest, or link-rot repair. Not for the archived page content itself, live-site crawling, submitting new captures, or full-text search of archived pages. --- # Wayback Machine Scraper Apify Actor `arman-bd/wayback-machine-scraper`. Give it hosts or URL prefixes and get one dataset record per capture the Internet Archive holds. It runs without credentials. This returns the archive's capture index, not the archived bytes: every row carries a replay link you can fetch separately for the content. ## When to use it - SEO migration audits: enumerate every URL a site ever published, then diff against the new sitemap to find what was never redirected. - Recovering a page that no longer exists, by finding its last successful capture and following the replay link. - Attack-surface research, where historical hostnames and paths often reveal staging, admin and API routes that are still reachable. - Content-change detection on a single page, collapsing on `digest` so every row is a real byte change rather than a re-capture. - Link-rot repair: for a dead outbound link, resolve the newest capture with a 200 status and rewrite the reference. ## When not to use it - The archived HTML, images or assets. You get index rows and a replay link; fetch the link yourself for bytes. - Crawling the live site. Nothing here touches the origin, and a URL in the index may have been gone for years. - Full-text search of archived pages. There is no content query, only URL matching and metadata filters. - Submitting a page for archiving, or anything that writes to the archive. - Guaranteed completeness. The index reflects what the archive happened to crawl, so absence of a URL is not evidence it never existed. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/wayback-machine-scraper').call({ urls: ['apify.com/blog'], matchType: 'prefix', fromDate: '20230101', toDate: '', filterStatus: ['200'], collapse: 'urlkey', maxResults: 1000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~wayback-machine-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"urls":["apify.com"],"matchType":"domain","filterStatus":["200"],"collapse":"urlkey","maxResults":500}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/wayback-machine-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `urls` | string[] | yes | | Bare hosts, URL prefixes or full URLs. The scheme and trailing slashes are stripped; the host is lower-cased and the path's case is preserved. Duplicates removed. | | `matchType` | string | no | `"prefix"` | `exact` is that one URL, `prefix` is it and everything beneath, `host` is every path on that exact host, `domain` is the host plus all its subdomains. Widest is slowest. | | `fromDate` | string | no | `""` | Earliest capture. `YYYYMMDD`, and `YYYY` or `YYYYMM` also work. Non-digits are stripped, so `2020-01-01` is accepted too. Fewer than four digits is ignored. | | `toDate` | string | no | `""` | Latest capture, same rules. | | `filterStatus` | string[] | no | `[]` | Archived HTTP status codes to keep. Only well-formed three-digit strings survive; anything else is dropped without a warning. Empty keeps every capture. | | `collapse` | string | no | `"urlkey"` | Collapse consecutive rows sharing a field. `urlkey` gives one row per unique URL, `timestamp:8` one row per URL per day, `digest` drops unchanged re-captures. Empty gives every raw capture. | | `maxResults` | integer | no | `1000` | Cap on **total** snapshots, split evenly across targets by rounding up. `0` means no cap. | **`matchType` and `collapse` together decide whether the run finishes at all.** The index is a cold scan, not a search engine, so an uncollapsed `matchType: "domain"` query against a busy host can enumerate millions of rows and will exhaust the run timeout long before it is done. `collapse: "urlkey"` is the right default for discovery because it turns millions of captures into one row per URL. Reach for `digest` only when you are watching a single page and want real changes, and leave `collapse` empty only on an `exact` target. Filters do not make the scan cheaper, they make the result smaller: `filterStatus`, `fromDate` and `toDate` all narrow what comes back per page, so pair a wide `matchType` with a real `maxResults` rather than trusting the filters to bound the work. ## Output One record per capture that survived collapsing and filtering. | Field | Type | Notes | |---|---|---| | `originalUrl` | string \| null | The URL exactly as it was crawled, including scheme and any query string. | | `timestamp` | string \| null | Raw Wayback timestamp, `YYYYMMDDhhmmss` in UTC. Sorts lexicographically as well as chronologically. | | `snapshotDate` | string \| null | The same instant as ISO 8601 with a `Z`. | | `archivedUrl` | string \| null | Permanent replay link built from the timestamp and the original URL. Fetch this for the bytes. `null` if either part was missing. | | `statusCode` | number \| null | The HTTP status the archive recorded. `null` for revisit and some redirect records, which carry a placeholder rather than a code. | | `mimeType` | string \| null | Content type as served at capture time. | | `digest` | string \| null | Base32 SHA-1 of the captured bytes. Identical digests mean identical content. | | `length` | number \| null | Compressed capture size in bytes. `null` where the index has no value. | | `urlkey` | string \| null | The canonical sort key the archive orders on, in reversed-host form, for example `com,apify)/blog`. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record: ```json { "originalUrl": "https://apify.com/", "timestamp": "20200109130705", "archivedUrl": "https://web.archive.org/web/20200109130705/https://apify.com/", "statusCode": 200, "mimeType": "text/html", "digest": "D4P2GPUK3PEGEFYQFUENG7MNTQI3ZRN4", "length": 29306, "urlkey": "com,apify)/", "snapshotDate": "2020-01-09T13:07:05Z", "scrapedAt": "2026-08-06T11:44:53.773Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "targetsRequested": 1, "targetsFailed": 0, "failures": [], "snapshotsSaved": 1000, "filters": { "matchType": "prefix", "fromDate": "20230101", "toDate": null, "filterStatus": ["200"], "collapse": "urlkey", "maxResults": 1000 }, "finishedAt": "2026-08-06T11:45:29.118Z" } ``` `snapshotsSaved` equal to `maxResults` almost always means you were truncated, not that the index is exhausted; raise the cap or narrow the query and compare. A target that matched nothing is **not** a failure: it leaves `failures` empty and contributes zero rows, so a run can report success with an empty dataset. When that happens, check `filters.matchType` first, because `exact` on a bare host matches only the bare homepage URL. `filters.fromDate` and `filters.toDate` show the normalised digit strings actually sent, and `filters.filterStatus` shows only the codes that survived validation, so compare it against what you passed. ## Behaviour to plan around - **Collapsing works on consecutive rows in the index's sort order.** That is why `urlkey` and `digest` are reliable, since the index is sorted by URL key then timestamp, and why collapsing on an arbitrary field is not. A `digest` collapse on a wide `prefix` query does not deduplicate across different URLs. - **`statusCode` is `null`, not zero, on revisit records.** The index writes a placeholder where a capture points at identical earlier bytes. Do not filter on `statusCode === 200` if you want those; use `filterStatus` instead, which is applied upstream. - **`maxResults` is a total, divided by rounding up.** Three targets with `maxResults: 1000` get 334 each, so the run can finish with 1002 rows. Do not treat the cap as exact. - **A target with no captures is a warning, not a failure.** The Actor throws only when every target errored and nothing was saved. `targetsFailed: 0` with zero rows means your query matched nothing. - **Malformed status codes are dropped silently.** `filterStatus: ["2xx", "200"]` filters on 200 only. Check `RUN_SUMMARY.filters.filterStatus`. - **Paging is cursor-based and self-limiting.** Pages are 1000 rows, and the run stops on an empty page, on a missing cursor, or when the index hands back a cursor it already gave, which is logged as a warning. That last case truncates the result without an error. - **Wide queries are slow before they are anything.** A `domain` query on a busy host routinely takes 30 to 60 seconds to return its first page. Requests allow 120 seconds and retry with exponential backoff, so a run that looks stalled is usually just scanning. - **A rejected query fails fast.** A 400 or 404 is treated as final for that target, no retries. Throttling and server faults get four attempts at 2, 4 and 8 seconds. - **Host case is normalised, path case is not.** `Example.com/Foo` becomes `example.com/Foo`, because paths can be case-sensitive and lower-casing them would miss captures. ## Recipes **Full URL inventory for a migration audit.** One row per unique URL across the whole domain including subdomains. ```json { "urls": ["apify.com"], "matchType": "domain", "collapse": "urlkey", "maxResults": 5000 } ``` Diff `originalUrl` against your new sitemap. Anything in the archive and not in the sitemap is a redirect you owe. **Successful pages from one section in a window.** ```json { "urls": ["apify.com/blog"], "matchType": "prefix", "fromDate": "20220101", "toDate": "20241231", "filterStatus": ["200"], "collapse": "urlkey", "maxResults": 2000 } ``` Use `archivedUrl` to fetch the bytes for anything you need to recover. **Change detection on one page.** One row per distinct version, oldest first by `timestamp`. ```json { "urls": ["https://example.com/pricing"], "matchType": "exact", "collapse": "digest", "maxResults": 500 } ``` Consecutive rows with different `digest` values are real edits. Re-run on a schedule and compare the newest `digest` against the one you stored. **Rule of thumb for sizing.** Start with `matchType: "host"`, `collapse: "urlkey"` and `maxResults: 1000`. If `snapshotsSaved` comes back at exactly the cap, the site is bigger than one run: raise the cap, or split the work by `fromDate` and `toDate` windows and merge on `originalUrl`.