--- name: certificate-transparency-scraper description: Enumerate every publicly logged TLS certificate for a domain and its subdomains via the Apify Actor arman-bd/certificate-transparency-scraper. Returns one record per certificate, or per unique host name in deduplication mode, with the subject alternative name list, issuing CA, serial number, validity window, expiry flag, CT log entry timestamp and a link to the certificate. Use for passive attack-surface mapping, forgotten staging-host discovery, CA and renewal-cadence audits and typosquat certificate monitoring. Not for live DNS resolution, port or service scanning, internal PKI, or any host that never had a publicly trusted certificate. --- # Certificate Transparency Scraper: Subdomain Discovery Apify Actor `arman-bd/certificate-transparency-scraper`. Give it domains, get the certificate transparency record for each one: every host name ever certified under it, who issued the certificate and when it was valid. It runs without credentials and never contacts the target domain, so the target sees nothing. ## When to use it - Mapping an organisation's external host names, including ones that were never linked to and no longer resolve. - Hunting decommissioned `staging.`, `dev.`, `uat.` and `internal.` hosts, which is what expired certificates are good for. - Auditing which CAs an estate uses and how often certificates are renewed. - Watching look-alike domains for a newly issued certificate, the usual last step before a phishing site goes live. - Producing a starting host list to feed into DNS resolution or an HTTP probe of your own. ## When not to use it - Deciding whether a host is live. CT proves a certificate existed, not that anything answers on that name today. - Internal PKI, self-signed certificates, or hosts that never had a publicly trusted certificate. None of those reach a CT log, so this is a floor on the attack surface and never a ceiling. - Port scans, service fingerprinting, content fetching or anything that touches the target. The Actor only reads a public append-only log. - WHOIS, registrant or DNS record data. Use a WHOIS or DNS Actor for those. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/certificate-transparency-scraper').call({ domains: ['apify.com'], includeExpired: true, deduplicateSubdomains: true, maxResultsPerDomain: 0, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request. Allow generous client-side time: a busy domain can take a minute on its own. ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~certificate-transparency-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"domains":["apify.com"],"includeExpired":false,"deduplicateSubdomains":true,"maxResultsPerDomain":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/certificate-transparency-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `domains` | string[] | yes | | Registrable domains or hosts. The match covers the domain and everything beneath it, so `apify.com` also returns `docs.apify.com`. Full URLs, `*.` wildcard prefixes, ports, trailing dots and mixed case are all normalised. Entries that do not normalise to a host name are dropped, and duplicates are removed. | | `includeExpired` | boolean | no | `true` | Keep certificates whose validity window has ended. Off is both narrower and faster, because the filter is applied at the source rather than locally. | | `deduplicateSubdomains` | boolean | no | `false` | Switches the record grain. Off, one record per certificate keyed on its common name. On, every subject alternative name is expanded and the first certificate covering each name is kept, giving one record per distinct host. | | `maxResultsPerDomain` | integer | no | `0` | Cap on records saved per domain, `0` for no limit. Certificates arrive newest first, so a low cap keeps the most recent activity. | **`deduplicateSubdomains` is the decision that changes the meaning of every row.** Off, the dataset is certificate history: renewals repeat, and that repetition is the signal for cadence and CA-change analysis. On, the dataset is a host inventory: one row per name, no renewals, and `nameValue` still carries the sibling names from the certificate that introduced it. Note that `maxResultsPerDomain` counts saved records, so the same number means "certificates" in one mode and "unique hosts" in the other. ## Output One record per certificate, or per unique host name when `deduplicateSubdomains` is on. | Field | Type | Notes | |---|---|---| | `domain` | string | The normalised input domain this record came from. Join back to your input list on this. | | `subdomain` | string \| null | The host name this record represents. The certificate's common name in default mode, or one expanded SAN in deduplication mode. | | `commonName` | string \| null | The certificate's own common name, lowercased. Present in both modes, so in deduplication mode it can differ from `subdomain`. | | `nameValue` | string[] | Every subject alternative name in the certificate, lowercased and de-duplicated. Includes wildcard entries such as `*.apify.com`. | | `issuerName` | string \| null | Full issuer distinguished name, for example `C=US, O=Let's Encrypt, CN=YR1`. Parse it if you want the O or CN separately. | | `issuerCaId` | number \| null | The source's internal CA identifier. Stable enough to group by, but not a public CA registry ID. | | `serialNumber` | string \| null | Hex serial, no separators. | | `notBefore` | string \| null | Start of the validity window. Naive UTC, for example `2026-08-04T11:02:40`, with no zone suffix. | | `notAfter` | string \| null | End of the validity window. Same naive UTC format. | | `isExpired` | boolean \| null | Computed against the run's clock from `notAfter`. `null` when `notAfter` is missing or unparseable. | | `entryTimestamp` | string \| null | When the CT log accepted the entry. Naive UTC with milliseconds. This, not `notBefore`, is the ordering key. | | `certificateId` | number \| null | The source's certificate identifier. Use it to detect newly issued certificates between runs. | | `certificateUrl` | string \| null | Direct link to the certificate detail page. `null` when there is no identifier. | | `scrapedAt` | string | Run timestamp, ISO 8601 with a `Z`. The only field here that is a true ISO 8601 instant. | A real record, from a run over `apify.com`: ```json { "domain": "apify.com", "subdomain": "odoo.apify.com", "commonName": "odoo.apify.com", "nameValue": ["odoo.apify.com"], "issuerName": "C=US, O=Let's Encrypt, CN=YR1", "issuerCaId": 432476, "serialNumber": "054316f0ae3d99286f3e349bbb02d2b651b7", "notBefore": "2026-08-04T11:02:40", "notAfter": "2026-11-02T11:02:39", "isExpired": false, "entryTimestamp": "2026-08-04T12:01:12.468", "certificateId": 28524761903, "certificateUrl": "https://crt.sh/?id=28524761903", "scrapedAt": "2026-08-06T11:38:30.148Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "domainsRequested": 2, "domainsFailed": 0, "certificatesFound": 968, "uniqueSubdomainsFound": 39, "recordsSaved": 39, "failures": [], "filters": { "domains": ["apify.com", "zzq-nonexistent-domain-4471.example"], "includeExpired": true, "deduplicateSubdomains": true, "maxResultsPerDomain": 40 }, "finishedAt": "2026-08-06T11:39:05.192Z" } ``` That run is real: 968 certificates collapse to 39 distinct host names, and the second domain has no certificates at all yet did not fail. `certificatesFound` far above `recordsSaved` is expected in deduplication mode and is not loss. Real loss shows up two ways: a non-empty `failures` array, each entry carrying `domain` and `error`, and `recordsSaved` sitting exactly on `maxResultsPerDomain` times the domain count, which means the cap truncated rather than the data running out. `uniqueSubdomainsFound` is `null` whenever `deduplicateSubdomains` is off. ## Behaviour to plan around - **This is slow, by nature of the source, and that is not a fault.** Budget roughly a minute per busy domain. There is also a deliberate five-second pause between domains, because back-to-back queries are what provoke failures in the first place. - **Availability is patchy and the Actor plans for it.** Up to six attempts per domain with five-second linear backoff. An overloaded source answers with an HTML error page rather than JSON, so the Actor decides on content type instead of status code. - **A domain with no certificates is a success, not a failure.** It records zero results and moves on. Only a domain that exhausted its retries lands in `failures`, and only a run where every domain failed throws. - **`notBefore`, `notAfter` and `entryTimestamp` have no timezone suffix.** They are UTC but naive, so appending `Z` before parsing is required. Passing them raw to a local date constructor silently shifts them by your offset. - **`isExpired` is computed at run time, not stored upstream.** A cached dataset goes stale: recompute from `notAfter` if you read the records days later. - **The result set arrives newest first, ordered by CT log entry.** `maxResultsPerDomain` therefore keeps recent activity, and there is no pagination to resume: the source returns everything in one response or nothing. - **Wildcard names appear as literal `*.` entries** in `nameValue` and can become the `subdomain` of a record in deduplication mode. Strip or expand them yourself before feeding the list to a resolver. - **One host renewed every 90 days produces dozens of certificates.** In the default mode that is intended. Counting rows is counting certificates, not hosts. - **`nameValue` can contain names outside the domain you asked for.** A shared or multi-tenant certificate lists every name it covers. Filter on suffix before treating the list as belonging to your target. ## Recipes **Full attack-surface inventory.** Widest possible host list, one row per name. ```json { "domains": ["example.com"], "includeExpired": true, "deduplicateSubdomains": true, "maxResultsPerDomain": 0 } ``` Filter `subdomain` for `staging`, `dev`, `test`, `uat`, `internal`, then resolve the survivors yourself to separate live hosts from historical ones. **Live estate audit.** Only names with a currently valid certificate, and noticeably faster because the expiry filter is applied at the source. ```json { "domains": ["example.com", "example.org"], "includeExpired": false, "deduplicateSubdomains": true, "maxResultsPerDomain": 0 } ``` **CA and renewal cadence audit.** Deduplication off, so the renewals you want to count are all present. ```json { "domains": ["example.com"], "includeExpired": true, "deduplicateSubdomains": false, "maxResultsPerDomain": 500 } ``` Group by `issuerName` for CA mix, and diff `notBefore` within each `commonName` for renewal interval. **Daily typosquat watch.** Recent issuance only, small cap, scheduled. ```json { "domains": ["exarnple.com", "example-support.com", "examp1e.com"], "includeExpired": false, "deduplicateSubdomains": false, "maxResultsPerDomain": 25 } ``` Alert on any `certificateId` not seen in the previous run.