--- name: dns-records-scraper description: Resolve DNS records for a list of domains via the Apify Actor arman-bd/dns-records-scraper. Returns one record per domain and record type, covering A, AAAA, MX, TXT, NS, CNAME, SOA and CAA, with response codes and record types decoded to names, the answer set structured with TTLs, mail exchangers sorted by preference, authoritative nameservers, and SPF and DMARC decoded into policy fields. Use when a task needs bulk DNS lookups, an email-spoofing audit, mail-provider detection, or a before-and-after diff of a domain's configuration. Not for reverse lookups, zone transfers, port scans, WHOIS registration data or certificate inspection. --- # DNS Records Scraper Apify Actor `arman-bd/dns-records-scraper`. Give it a list of hostnames, get one dataset record per domain and record type, resolved over DNS-over-HTTPS through Google or Cloudflare. It takes no credentials of your own. Numeric response codes and record types arrive decoded: `NXDOMAIN`, not `3`; `MX`, not `15`. ## When to use it - Auditing email security across a customer or prospect list: SPF and DMARC in one pass. - Detecting a company's mail or hosting provider in bulk from `mxHosts` and `value`. - Capturing a domain's full configuration before a migration, then diffing after. - Watching a domain list on a schedule for changed records, expired entries or a sub-domain that has started pointing somewhere new. - Checking which certificate authorities a domain permits, through `CAA`. ## When not to use it - Reverse DNS, PTR sweeps or zone transfers. Forward lookups of named types only. - Registration data: registrar, creation date, expiry, registrant. That is a different Actor and a different protocol. - Anything that touches the hosts themselves. Nothing here connects to a server, opens a port or fetches a page. - Certificate contents or TLS configuration. `CAA` tells you what is *permitted*, not what was issued. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/dns-records-scraper').call({ domains: ['apify.com', 'stripe.com', 'github.com'], recordTypes: ['MX', 'TXT'], resolver: 'google', parseEmailPolicy: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~dns-records-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"domains":["apify.com","stripe.com"],"recordTypes":["MX","TXT"],"parseEmailPolicy":true}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/dns-records-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `domains` | string[] | yes | | Hostnames. Schemes, credentials, paths, ports, trailing dots and email-style inputs are all stripped, so `apify.com`, `https://apify.com/store` and `user@apify.com` all become `apify.com`. Sub-domains resolve as written. Lower-cased, deduplicated. Anything that is not a hostname is rejected and reported, not queried. | | `recordTypes` | string[] | no | `["A","AAAA","MX","TXT","NS"]` | Any of `A`, `AAAA`, `MX`, `TXT`, `NS`, `CNAME`, `SOA`, `CAA`. Case-insensitive, deduplicated. One lookup and one dataset record each. Unsupported values are logged and dropped; an empty valid set throws. | | `resolver` | string | no | `"google"` | `google` or `cloudflare`. Whichever you pick answers first and the other is the automatic fallback. An unrecognised value falls back to `google`. | | `parseEmailPolicy` | boolean | no | `true` | Extract SPF from the domain's TXT records and fetch DMARC from the `_dmarc` sub-domain, then decode both. Off removes the six email fields from the output entirely. | **Cost is `domains × recordTypes`, and `parseEmailPolicy` adds to it.** Trimming `recordTypes` is the only real lever on a large sweep. Email parsing costs one extra lookup per domain for DMARC, plus a second one for SPF only when `TXT` is not already in `recordTypes`. So `recordTypes: ["MX","TXT"]` with `parseEmailPolicy: true` is three lookups per domain and is the efficient shape for an email audit; the same setting with `recordTypes: ["MX"]` costs four. ## Output One record per domain and record type that returned an answer, successful or not. | Field | Type | Notes | |---|---|---| | `domain` | string | The normalised hostname that was queried. | | `recordType` | string | The type this lookup asked for. | | `status` | string | Decoded response code: `NOERROR`, `NXDOMAIN`, `SERVFAIL`, `REFUSED` and the rest. Unknown codes come back as `RCODE`. | | `statusCode` | number | The raw integer behind `status`. | | `value` | string[] | Every answer flattened to its data string. The field to eyeball. | | `records` | object[] | The same answers structured: `{ name, type, ttl, data }`, type already decoded. | | `ttl` | number \| null | Shortest TTL in the answer set, in seconds. `null` when there are no answers. | | `resolver` | string | Which provider actually answered, `google` or `cloudflare`. | | `failedOver` | boolean | `true` when the fallback resolver answered because the primary failed. | | `responseTimeMs` | number | Round-trip time for this lookup. | | `hasSpf` | boolean | Present only when `parseEmailPolicy` is on. Whether an SPF record was found. | | `spfRecord` | string \| null | The full SPF record. Present only when `parseEmailPolicy` is on. | | `spfPolicy` | string \| null | The enforcement qualifier: `-all` hard fail, `~all` soft fail, `?all` neutral, `+all` pass-anything. `null` when SPF exists without an `all` mechanism. | | `hasDmarc` | boolean | Present only when `parseEmailPolicy` is on. | | `dmarcRecord` | string \| null | The full DMARC record. Present only when `parseEmailPolicy` is on. | | `dmarcPolicy` | string \| null | `none`, `quarantine` or `reject`, lower-cased from the record's `p=` tag. | | `mxHosts` | object[] | `{ preference, host }` for the domain, sorted by preference. Domain-level, so repeated on every row for that domain. Empty when `MX` was not requested. | | `nameservers` | string[] | Authoritative nameservers, trailing dots stripped. Domain-level and repeated. Empty when `NS` was not requested. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record: ```json { "domain": "apify.com", "recordType": "MX", "status": "NOERROR", "statusCode": 0, "value": [ "1 aspmx.l.google.com.", "5 alt1.aspmx.l.google.com.", "10 aspmx2.googlemail.com." ], "records": [ { "name": "apify.com", "type": "MX", "ttl": 86400, "data": "1 aspmx.l.google.com." } ], "ttl": 86400, "resolver": "google", "failedOver": false, "responseTimeMs": 33, "hasSpf": true, "spfRecord": "v=spf1 a mx include:_spf.google.com include:mailgun.org -all", "spfPolicy": "-all", "hasDmarc": true, "dmarcRecord": "v=DMARC1; p=reject; sp=reject; pct=100; rua=mailto:dmarc-reports@apify.com; ri=604800", "dmarcPolicy": "reject", "mxHosts": [ { "preference": 1, "host": "aspmx.l.google.com" }, { "preference": 5, "host": "alt1.aspmx.l.google.com" }, { "preference": 10, "host": "aspmx2.googlemail.com" } ], "nameservers": [ "ns-449.awsdns-56.com", "ns-839.awsdns-40.net", "ns-1225.awsdns-25.org", "ns-1928.awsdns-49.co.uk" ], "scrapedAt": "2026-08-06T11:43:11.221Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "domainsRequested": 2, "domainsRejected": ["not a domain!!"], "lookupsRequested": 2, "lookupsFailed": 0, "failures": [], "lookupsSaved": 2, "filters": { "domains": ["apify.com", "stripe.com"], "recordTypes": ["MX"], "resolver": "google", "parseEmailPolicy": true }, "finishedAt": "2026-08-06T11:43:48.478Z" } ``` `domainsRequested` counts what survived normalisation, not what you sent, so check `domainsRejected` for typos before concluding a domain has no records. `lookupsRequested` is `domainsRequested × recordTypes` and excludes the extra SPF and DMARC lookups. `lookupsSaved` short of `lookupsRequested` is explained entirely by `failures`, which names the domain and record type for each. ## Behaviour to plan around - **Absence of a record is not absence of a domain.** `NOERROR` with an empty `value` means the domain exists but has nothing of that type, which is normal for `CNAME` on an apex and for `AAAA` on IPv4-only hosts. `NXDOMAIN` means the name does not exist at all. `SERVFAIL` usually means a real misconfiguration or a DNSSEC failure and is worth flagging rather than retrying. - **When `parseEmailPolicy` is off, the six email fields are missing, not `null`.** Code that reads `record.hasSpf` will see `undefined`. Check the flag in `RUN_SUMMARY.filters` before treating an absent field as a finding. - **`value` and `records` carry the whole answer chain.** A query for `A` on a host that is a CNAME returns the CNAME entry and the A entry. Filter `records` on `type === recordType` if you want only what you asked for. `mxHosts` and `nameservers` are already filtered this way. - **`mxHosts` and `nameservers` are domain-level and repeated on every row** for that domain, so one row answers "who runs this domain's mail". They are empty unless `MX` or `NS` respectively were in `recordTypes`. - **The two resolvers can legitimately disagree.** Geo-aware and load-balanced DNS answers depend on who asked. That is the state of the world, not an error. Pin `resolver` if you need runs to be comparable, and treat `failedOver: true` rows as answered by the other provider. - **TXT values are joined before comparison.** Long records are split into segments in the wire format and the two providers hand them back differently. The Actor concatenates them, which is what makes diffing `value` across scheduled runs meaningful. - **TTL is the minimum of the answer set,** because the shortest TTL governs when the whole answer goes stale. - **One failed lookup never kills the run.** Retries are three attempts with linear backoff; a malformed request or an unexpected shape fails immediately. Failures land in `RUN_SUMMARY.failures` and the Actor only throws when nothing at all was saved. - **`spfPolicy` can be `null` while `hasSpf` is `true`.** A record without an `all` mechanism has no enforcement qualifier to report. Do not read `null` as "no SPF". - **DMARC lives on the `_dmarc` sub-domain, not the domain.** A sender with no DMARC record gets `hasDmarc: false`, and that absence is itself the audit finding. ## Recipes **Email-spoofing audit across a domain list.** The efficient shape: TXT is in the list, so SPF costs nothing extra. ```json { "domains": ["apify.com", "stripe.com", "github.com"], "recordTypes": ["MX", "TXT"], "parseEmailPolicy": true } ``` Take one row per domain and sort on `dmarcPolicy`. Anything at `none` or `null`, or an `spfPolicy` of `?all` or `+all`, is a spoofing risk. **Mail-provider enrichment.** One lookup per domain, no policy parsing. ```json { "domains": ["stripe.com", "shopify.com", "notion.so"], "recordTypes": ["MX"], "parseEmailPolicy": false } ``` Match `mxHosts[].host` against known suffixes to label each domain's provider. Remember the email fields will be absent from these rows. **Pre-migration snapshot.** The full configuration picture, pinned to one resolver so the after-run is comparable. ```json { "domains": ["example.com", "www.example.com", "api.example.com"], "recordTypes": ["A", "AAAA", "CNAME", "NS", "SOA", "CAA"], "resolver": "cloudflare", "parseEmailPolicy": false } ``` Store the result, cut over, run it again, then diff `value` keyed on `domain` plus `recordType`. **Daily drift watch.** Everything that usually changes, nothing that does not. ```json { "domains": ["example.com", "shop.example.com"], "recordTypes": ["A", "CNAME", "NS", "TXT"], "parseEmailPolicy": true } ``` Alert on any change to `value`, and on `status` moving away from `NOERROR`. A sub-domain whose `CNAME` starts pointing at an unfamiliar host is the classic takeover signal.