--- name: website-tech-audit-scraper description: Audit the public technical files and technology stack of any list of sites via the Apify Actor arman-bd/website-tech-audit-scraper. Returns one record per site with parsed robots.txt rules, declared sitemaps and URL counts, ads.txt seller counts, security.txt contacts, homepage title, description and Open Graph tags, JSON-LD types, and a technology list where every detection carries the matched text as evidence. Use when a task needs bulk technical SEO audits, replatform checks, lead qualification by platform, or passive infrastructure reconnaissance. Not for page crawling, performance metrics, accessibility scoring or vulnerability scanning. --- # Website Tech Audit Scraper Apify Actor `arman-bd/website-tech-audit-scraper`. Give it domains, get one record per site describing what that site publishes for machines and what it appears to run. Every technology claim carries the text that proved it, so you can audit a detection without re-running the audit. The Actor takes no credentials. ## When to use it - Bulk technical SEO audits: crawlability, sitemap coverage and meta hygiene across a portfolio. - Pre- and post-migration checks: confirm what a site declares before and after a replatform, and diff `detectedPlatform` and `robotsRules` between runs. - Lead qualification: filter a prospect list down to sites running a platform you integrate with, with evidence attached for each. - Passive reconnaissance: security contacts, hosting and CDN signals, without probing anything beyond documented well-known paths and the homepage. ## When not to use it - Crawling a site's pages. Only the homepage and the well-known files are read; a sitemap's URLs are counted, never fetched. - Performance, Core Web Vitals, accessibility or Lighthouse-style scoring. - Vulnerability scanning, port scanning or anything behind authentication. - Exhaustive stack detection. The signal table covers roughly 60 technologies, and a page that renders entirely client-side may match none of them. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/website-tech-audit-scraper').call({ sites: ['nytimes.com', 'woocommerce.com', 'vercel.com'], checks: ['robots', 'sitemap', 'ads', 'security', 'meta', 'schema', 'tech'], expandSitemaps: false, maxUrlsPerSitemap: 1000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~website-tech-audit-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sites":["nytimes.com","woocommerce.com"],"checks":["robots","tech","meta"]}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/website-tech-audit-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `sites` | string[] | yes | | Domains or URLs. A bare domain, a hostname with www, or a full page URL all work. The scheme defaults to https and the path is stripped, since every file audited lives at the origin. A value with no dot in the hostname is dropped. Duplicates removed. An empty result aborts the run. | | `checks` | string[] | no | all seven | Any of `robots`, `sitemap`, `ads`, `security`, `meta`, `schema`, `tech`. Fewer checks means fewer requests. An empty array falls back to the three file checks only. | | `expandSitemaps` | boolean | no | `false` | Follow every discovered sitemap, walk indexes into their children, and count the URLs inside. One request per sitemap file. | | `maxUrlsPerSitemap` | integer | no | `1000` | Stop counting a site's sitemap URLs once this many have been seen. Only read when `expandSitemaps` is on. | **`checks` is a request budget, and it is not linear.** Each of `robots`, `ads` and `security` is one request. `sitemap` is zero or one, since it only probes when robots.txt declared no sitemap. `meta`, `schema` and `tech` share a single homepage fetch, so enabling all three costs exactly what enabling one costs. A full audit is therefore about five requests per site, and dropping to `["tech"]` saves four of them. `expandSitemaps` is the one option that can multiply the count without limit: a news publisher with a sitemap index of 200 children is 200 extra requests, capped only by how quickly `maxUrlsPerSitemap` is reached. ## Output One record per site, always carrying every field. A field left at its default means the check was disabled, which is why `robotsFound` is three-valued. | Field | Type | Notes | |---|---|---| | `site` | string | The normalised origin, scheme and host only. The join key. | | `robotsFound` | boolean \| null | `null` when the check was off, `false` when nothing usable was served. | | `robotsRules` | object[] | Every user-agent group, keys `userAgents`, `allow`, `disallow`, `crawlDelay`. Consecutive user-agent lines share one group, as the specification requires. | | `crawlDelay` | number \| null | From the wildcard group only. | | `disallowedPaths` | string[] | The wildcard group's disallow list, surfaced separately because that is what a generic crawler obeys. | | `sitemapUrls` | string[] | Sitemaps declared in robots.txt, plus children discovered inside sitemap indexes when expanding. | | `sitemapUrlCount` | number \| null | URLs counted inside those sitemaps. `null` unless `expandSitemaps` is on. Capped at `maxUrlsPerSitemap`. | | `adsFound` | boolean \| null | | | `adsSellerCount` | number \| null | Authorised seller lines. `null` when the file was not found. | | `securityTxtFound` | boolean \| null | | | `securityContacts` | string[] | The `Contact:` values, verbatim, usually a mailto or a URL. | | `metaTitle` | string \| null | Homepage title, entities decoded. | | `metaDescription` | string \| null | | | `ogTags` | object | Every `og:*` property found, keyed by the full property name. `{}` when none. | | `schemaTypes` | string[] | Unique JSON-LD type values, sorted. Malformed blocks still contribute. | | `detectedPlatform` | string \| null | The single best-supported CMS, e-commerce or builder detection. Never a CDN, framework or web server. | | `generator` | string \| null | Raw generator meta content, several joined with a semicolon. | | `technologies` | object[] | Keys `technology`, `category`, `confidence`, `evidence`. `confidence` is `strong` or `weak`; `evidence` is an array of `haystack: matched text` strings, truncated at 120 characters each. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record: ```json { "site": "https://woocommerce.com", "robotsFound": true, "robotsRules": [ { "userAgents": ["*"], "allow": [], "disallow": ["/wp-content/uploads/wc-logs/"], "crawlDelay": null } ], "crawlDelay": null, "disallowedPaths": ["/wp-content/uploads/wc-logs/", "/*?add-to-cart="], "sitemapUrls": ["https://woocommerce.com/sitemap.xml", "https://woocommerce.com/post-sitemap.xml"], "sitemapUrlCount": 200, "adsFound": false, "adsSellerCount": null, "securityTxtFound": false, "securityContacts": [], "metaTitle": "WooCommerce", "metaDescription": "WooCommerce is a customizable, open-source ecommerce platform built on WordPress.", "ogTags": { "og:type": "website", "og:site_name": "WooCommerce" }, "schemaTypes": ["BreadcrumbList", "Organization", "SearchAction"], "detectedPlatform": "WooCommerce", "generator": "WordPress 7.0.2; WooCommerce 11.1.0", "technologies": [ { "technology": "WordPress", "category": "CMS", "confidence": "strong", "evidence": ["generator: WordPress 7.0.2"] }, { "technology": "WooCommerce", "category": "E-commerce", "confidence": "strong", "evidence": ["generator: WooCommerce"] }, { "technology": "Nginx", "category": "Web server", "confidence": "strong", "evidence": ["header: server: nginx"] } ], "scrapedAt": "2026-08-06T12:00:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it**, and in particular read `partialAudits`, which is the only signal that a saved record is missing half its fields. ```json { "sitesRequested": 3, "sitesFailed": 0, "failures": [], "sitesAudited": 3, "partialAudits": [{ "site": "https://example.com", "error": "homepage unavailable (HTTP 403)" }], "filters": { "checks": ["robots", "sitemap", "ads", "security", "meta", "schema", "tech"], "expandSitemaps": false, "maxUrlsPerSitemap": 1000 }, "finishedAt": "2026-08-06T12:00:07.221Z" } ``` There are two distinct kinds of incompleteness here. `failures` holds sites that produced no record at all, one `{ site, error }` each, typically a DNS failure or an unreachable host; `sitesAudited` plus `sitesFailed` equals `sitesRequested`. `partialAudits` holds sites that **are** in the dataset but whose homepage could not be read, so `metaTitle`, `schemaTypes`, `technologies` and `detectedPlatform` are empty for them while the file-based findings are real. Treating a partial row's empty `technologies` as "this site runs nothing" is the single easiest mistake to make here. ## Behaviour to plan around - **`null` and `false` mean different things throughout.** `robotsFound: null` means the check was not enabled; `false` means it ran and found nothing usable. The same applies to `adsFound` and `securityTxtFound`, and `sitemapUrlCount: null` means you did not ask for expansion. - **Weak signals never decide `detectedPlatform` alone.** A page can carry a matcher without running the technology: one image served from a platform's asset path is enough to match. Such signals appear in `technologies` with `confidence: "weak"` and their evidence, but the platform field stays `null` unless a strong signal fired or at least two independent weak matchers agree. A newspaper that embeds a single logo from a WordPress asset path is reported as weak WordPress with no platform. - **`detectedPlatform` prefers the most specific category.** E-commerce beats website builder beats CMS, so a WooCommerce store reports WooCommerce rather than WordPress even though both are in `technologies`. - **Empty `technologies` is not evidence of a bare site.** Some homepages return a minimal shell or an error to a plain HTTP client, and there is nothing in that response to match. Cross-check `partialAudits` before drawing a conclusion. - **A file that returns HTML does not count as found.** Sites frequently answer a missing well-known file with a themed error page at a success status. Both the status and the shape of the body are checked, so a large HTML document is never reported as an ads.txt with hundreds of sellers. - **`adsSellerCount` excludes variable lines.** Declarations such as an owner-domain line are not authorised sellers. A site can legitimately publish the file with zero sellers to say that nobody may sell its inventory, so `adsFound: true` with `adsSellerCount: 0` is meaningful rather than broken. - **robots.txt is authoritative for sitemaps.** The conventional root sitemap path is probed only when robots.txt declared none, so `sitemapUrls` reflects what the site actually publishes rather than what it happens to host. - **Gzipped sitemaps are listed but not expanded.** They are served as raw compressed bytes that cannot be read as text, so they appear in `sitemapUrls` and contribute nothing to `sitemapUrlCount`. A low count against a large site usually means this. - **`sitemapUrlCount` is a floor, not a total.** It stops at `maxUrlsPerSitemap`, and index children are queued breadth-first, so the cap can be reached before every child is read. - **One bad site never aborts the run.** Failures are recorded and the rest continue; the Actor only throws when every site failed. Transient errors and network faults get three attempts with exponential backoff and a 30-second timeout. ## Recipes **Bulk technical SEO audit.** Crawlability and meta hygiene, no sitemap expansion so the run stays predictable. ```json { "sites": ["client-a.com", "client-b.com", "client-c.com"], "checks": ["robots", "sitemap", "meta", "schema"], "expandSitemaps": false } ``` Flag rows where `disallowedPaths` blocks something important, `metaDescription` is `null`, or `sitemapUrls` is empty. **Lead qualification by platform.** One request per site, the cheapest possible run. ```json { "sites": ["store-a.com", "store-b.com", "store-c.com"], "checks": ["tech"] } ``` Filter on `detectedPlatform`. For anything you plan to act on, read the matching entry in `technologies` and check its `evidence` is a header or generator rather than a single asset URL. **Sitemap coverage check before a migration.** ```json { "sites": ["example.com"], "checks": ["robots", "sitemap"], "expandSitemaps": true, "maxUrlsPerSitemap": 5000 } ``` Compare `sitemapUrlCount` before and after the replatform. If it drops, check whether `sitemapUrls` now lists gzipped files, which are counted as zero. **Passive contact and infrastructure sweep.** ```json { "sites": ["target-a.com", "target-b.com"], "checks": ["security", "tech"] } ``` Collect `securityContacts` for disclosure routing, and keep the `Hosting` and `CDN` entries from `technologies` for the infrastructure picture.