--- name: wordpress-blog-scraper description: Read posts, pages or any public custom post type from WordPress sites as structured records via the Apify Actor arman-bd/wordpress-blog-scraper. Returns one record per post with title, excerpt and full body as clean plain text, author id and display name, category and tag slugs, published and modified timestamps in UTC, featured image, comment count where countable, and the public link, across many sites in one run with server-side category, full-text and date filtering. Use for blog migration and archiving, competitor content tracking, multi-site content indexing or scheduled newsletter sourcing. Not for drafts, private or password-protected content, comment bodies, admin data, or sites that do not serve the public REST collections. --- # WordPress Blog Scraper Apify Actor `arman-bd/wordpress-blog-scraper`. Give it site domains, get one dataset record per post. It reads the public REST collections a stock WordPress install serves, so self-hosted sites and hosted ones behave the same. It runs without credentials; you supply nothing but the list of sites. ## When to use it - Exporting a whole blog to structured JSON before a replatform, or for an archive. - Tracking what competitors publish, how often and under which categories. - Building one searchable index across dozens of publisher sites, with the same schema for each. - A scheduled pull of new posts for a newsletter or aggregator, bounded by date. - Extracting body text for a retrieval corpus without writing an HTML parser. ## When not to use it - Drafts, scheduled, private or password-protected content. Only what the site publishes publicly is visible. - Comment bodies, users, settings, plugins or anything behind the admin. A comment count is as far as it goes. - Sites that do not serve the public REST collections. Those are recorded as failures, not worked around. - Non-WordPress sites, and WordPress sites you want scraped as rendered HTML. This reads structured data, not pages. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/wordpress-blog-scraper').call({ sites: ['wordpress.org/news', 'techcrunch.com'], postType: 'posts', categories: [], sinceDate: '2026-01-01', maxPostsPerSite: 100, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~wordpress-blog-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"sites":["wordpress.org/news"],"categories":["releases"],"maxPostsPerSite":25}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/wordpress-blog-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `sites` | string[] | yes | | A bare domain, a sub-directory install (a domain plus a path segment), or a full article URL, which is reduced to its origin. A scheme is added if missing, a pasted REST route suffix is stripped, and duplicates are removed. An entry whose hostname has no dot is discarded. | | `postType` | string | no | `"posts"` | Which collection to read: posts, pages, or any public custom post type the site has registered for REST access, named by its REST base. Surrounding slashes are stripped. | | `categories` | string[] | no | `[]` | Category slugs, as they appear in a category archive URL. Resolved to term ids before filtering, because the underlying parameter ignores slugs silently. Applies to the posts type only. | | `searchQuery` | string | no | `""` | Full-text search run by the site itself across title and body. | | `sinceDate` | string | no | `""` | Only posts published after this. A plain date or a full ISO 8601 timestamp. An unparseable value is warned about and ignored, not fatal. | | `maxPostsPerSite` | integer | no | `100` | Posts saved per site. `0` means the entire archive, which for a large publisher is hundreds of thousands of posts. | **The three filters run on the site, not here, and they combine with AND.** That is what makes this cheap: `categories`, `searchQuery` and `sinceDate` cut the result set before anything is transferred, so a narrow query over a huge publisher costs a few requests rather than a full crawl. `maxPostsPerSite` is the opposite kind of limit, a stop condition applied while paging, so it bounds run time but tells you nothing about how much matched. Set the filters as tightly as the question allows and treat `maxPostsPerSite` as a safety rail rather than as a selection tool. ## Output One record per post. | Field | Type | Notes | |---|---|---| | `site` | string | Normalised origin the record came from, including a sub-directory path where the install has one. Join back on this. | | `postId` | number | The site's own post id. Unique within a site, not across sites, so key on `site` plus `postId`. | | `title` | string | Entity-decoded title. Inline markup, if the site uses any, is left in place. | | `slug` | string \| null | URL slug. | | `excerpt` | string | Excerpt as plain text. Empty string when the site publishes none. | | `contentPlain` | string | Full body as plain text: HTML stripped, entities decoded twice for double-encoding plugins, list items prefixed with a bullet, blank runs collapsed. | | `status` | string \| null | As the site reports it, normally publish. | | `author` | number \| null | Numeric author id. Always present when the site sets one. | | `authorName` | string \| null | Display name. `null` when the site does not serve its users collection, or when authorship is attributed by a plugin. | | `categories` | string[] | Category **slugs**, resolved from term ids. See the warning below about empty arrays. | | `tags` | string[] | Tag slugs, resolved the same way. | | `publishedAt` | string \| null | UTC ISO 8601 with an explicit zone. Falls back to the site-local timestamp, without a zone, when the UTC field is absent. | | `modifiedAt` | string \| null | Same handling as `publishedAt`. The field to diff on for updates. | | `featuredImage` | string \| null | Full-size image URL, resolved from the media collection, falling back to a hosted-platform image field. | | `commentCount` | number \| null | Tallied from the comments collection. `null` means it could not be counted, never a substitute for zero. | | `link` | string \| null | Public URL of the post. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "site": "https://wordpress.org/news", "postId": 21248, "title": "WordPress 7.1 Release Candidate 1", "slug": "wordpress-7-1-release-candidate-1", "excerpt": "The first Release Candidate (\"RC1\") for WordPress 7.1 is ready for download and testing!", "contentPlain": "The first Release Candidate (\"RC1\") for WordPress 7.1 is ready…", "status": "publish", "author": 14679285, "authorName": "Benjamin Zekavica", "categories": ["development", "general", "releases"], "tags": ["7-1", "development", "release"], "publishedAt": "2026-08-05T15:37:33Z", "modifiedAt": "2026-08-05T15:37:33Z", "featuredImage": "https://wordpress.org/news/files/2026/07/wordpress-7-1-RC1.png", "commentCount": 0, "link": "https://wordpress.org/news/2026/08/wordpress-7-1-release-candidate-1/", "scrapedAt": "2026-08-06T12:00:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** Sites vary enormously in what they expose, and this is where a site that gave you nothing says so. ```json { "sitesRequested": 3, "sitesFailed": 1, "failures": [ { "site": "https://example-blog.com", "error": "non-JSON response (text/html), REST API likely disabled" } ], "postsSaved": 217, "filters": { "postType": "posts", "categories": ["releases"], "searchQuery": "", "sinceDate": "2026-01-01T00:00:00", "maxPostsPerSite": 100 }, "finishedAt": "2026-08-06T12:00:19.774Z" } ``` `failures[].error` distinguishes the cases that matter: a route that does not exist, a response that was not JSON at all (which is what a site with the REST collections turned off usually returns), a rejected query (normally a bad `postType`), and a category filter that matched nothing on that site. `filters.sinceDate` is the **normalised** value, so `null` there after you passed something means the date was unparseable and the whole archive was read. `postsSaved` at exactly `sitesRequested` times `maxPostsPerSite` means every site hit the cap and there is more behind it. ## Behaviour to plan around - **An empty `categories` array does not mean the post is uncategorised.** Term slugs are resolved through a separate lookup, and a site that does not serve its terms collection leaves both `categories` and `tags` empty for every post with only a warning in the log. Check whether the arrays are empty across the whole site before drawing a conclusion. - **`commentCount: null` is deliberate.** It means the comments collection could not be read, or held more comments than the page budget could tally. A fabricated zero would be worse. Never sum a column containing nulls without excluding them. - **`authorName` is null on a good share of sites** while `author` is still a valid id. If you need names on such a site, map the ids yourself; the id is stable within the site. - **The category filter applies to the posts type only.** With any other `postType` it is ignored and a warning is logged, so you get the whole collection rather than an error. - **A category slug that exists nowhere on a site fails that site.** If none of the requested slugs resolve, the site is recorded in `failures`. If some resolve, the unknown ones are logged and skipped, and you get a narrower set than you asked for. - **`publishedAt` is not always UTC.** It carries a trailing zone marker when the site publishes a UTC timestamp, which is almost always, and falls back to a zone-less site-local string when it does not. Parse defensively before comparing across sites. - **Timestamps, not ids, are the change signal.** Post ids are per-site and never reissued, but an edited post keeps its id, so diff on `postId` plus `modifiedAt`. - **One bad site never aborts the run.** A missing route, a non-JSON response or a DNS failure is recorded in `failures` and the next site is read. The Actor only throws when every site failed. - **A themed error page returned with a success status is rejected.** The response content type is checked, so a site that answers with HTML is treated as unavailable rather than parsed into empty rows. - **Transient errors are retried** three times with exponential backoff. A missing route, a rejected query and a non-JSON body are final for that site. - **Paging is read, not guessed**, from the pagination metadata the site returns, and the page size is fixed for the whole site at up to 100 posts per request. - **Pass `maxPostsPerSite` explicitly.** The schema default is 100, but if the key is absent from the raw input the Actor falls back to no cap, which on a large publisher is a very long run. ## Recipes **Two sites, most recent posts.** The simplest useful run. ```json { "sites": ["wordpress.org/news", "techcrunch.com"], "postType": "posts", "maxPostsPerSite": 100 } ``` Sort on `publishedAt` descending. Rows from different sites are directly comparable because the schema is identical. **One category, this year, capped.** Filtering happens on the site, so this is cheap even on a large archive. ```json { "sites": ["wordpress.org/news"], "categories": ["releases"], "sinceDate": "2026-01-01", "maxPostsPerSite": 50 } ``` Confirm in `RUN_SUMMARY.failures` that the site did not reject the slug before reporting a count. **Full archive export before a migration.** No cap, no filters. ```json { "sites": ["wordpress.org/news"], "postType": "posts", "maxPostsPerSite": 0 } ``` `contentPlain` is body text, not HTML, so keep `link` if you will need to re-fetch the original markup. Run a second time with the pages type to capture static pages. **Scheduled new-posts feed.** Date-bounded, so each run is small. ```json { "sites": ["wordpress.org/news", "techcrunch.com"], "sinceDate": "2026-08-01T00:00:00", "maxPostsPerSite": 200 } ``` Set `sinceDate` to the previous run's finish time and deduplicate on `site` plus `postId`; an unchanged `modifiedAt` means the post is a repeat, not an edit.