--- name: substack-newsletter-scraper description: Read recent posts from any public Substack publication as structured records via the Apify Actor arman-bd/substack-newsletter-scraper. Each record carries the publication, post id and canonical link, title, subtitle, author, publish time, the full body as clean plain text and optionally as original markup, a word count, cover image, podcast audio URL and a paywalled-preview flag. Use when a task needs newsletter monitoring, competitor or thought-leader tracking, a long-form text corpus, or new-post alerts on a schedule. Not an archive tool: it returns only the 20 most recent posts per publication and never subscriber-only text. --- # Substack Scraper: Newsletter Posts Apify Actor `arman-bd/substack-newsletter-scraper`. Give it publication slugs, domains or post links, get one dataset record per post from each publication's public feed. The body of every post is inlined in that feed, so a single request per publication returns full text. You supply no credentials. ## When to use it - Monitoring a shortlist of newsletters and picking up whatever is new since last run. - Competitive tracking: what rivals ship, how often, and how long their posts are. - Building a long-form text corpus for analysis, retrieval or summarisation. - Alerting on a topic across many publications, matching on `title` and `contentText`. - Indexing posts by author and subtitle for a discovery or recommendation feed. ## When not to use it - Full archives. Each feed carries only the 20 most recent posts, and there is no pagination. To go further back you need a different source. - Subscriber-only text. Paid posts arrive as the free preview, flagged, and nothing here retrieves what is behind the paywall. - Comments, likes, subscriber counts, revenue or any Substack engagement metric. None of it is in the feed. - Substack Notes, chat threads or podcast transcripts. Podcast posts do return an `audioUrl`, but only the enclosure, never the audio itself. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/substack-newsletter-scraper').call({ publications: ['astralcodexten', 'newsletter.pragmaticengineer.com'], sinceDate: '2026-08-01', maxPostsPerPublication: 10, includeContent: true, stripHtmlTags: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~substack-newsletter-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"publications":["astralcodexten","noahpinion"],"maxPostsPerPublication":5,"includeContent":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/substack-newsletter-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `publications` | string[] | yes | | A bare slug (`astralcodexten`), a `{slug}.substack.com` URL, a custom domain (`newsletter.pragmaticengineer.com`), or a link to any single post or archive page. Anything containing a dot is treated as a host. Mixed forms are fine. Values that cannot be resolved are dropped with a log warning only. | | `maxPostsPerPublication` | integer | no | `0` | Cap on saved posts per publication, `0` to `20`, applied **after** the date filter. `0` means everything the feed carries. Above 20 has no effect. | | `sinceDate` | string | no | `""` | Keep only posts published at or after this moment. A plain date (`2026-01-01`) or a full ISO 8601 timestamp. An unparseable value throws before any request. | | `includeContent` | boolean | no | `true` | Put the post body in `contentText`. Off leaves it `null` and shrinks the dataset by orders of magnitude. | | `stripHtmlTags` | boolean | no | `true` | On, bodies are plain text and `contentHtml` is `null`. Off additionally keeps the original markup in `contentHtml`, which roughly doubles the record and is what you want for inline images, links and footnotes. | **`includeContent` is the only expensive decision.** Cost in requests is fixed at one per publication whatever you set, but the feed inlines every post body, so a busy publication is a couple of megabytes of text either way. What the flag controls is how much of that ends up in your dataset. For scheduled monitoring, pair `sinceDate` with `includeContent: false` and fetch bodies in a second, narrower run. Note that `wordCount` is computed before the flag is applied, so you keep the length signal even with content off. ## Output One record per post kept, in feed order, which is newest first. | Field | Type | Notes | |---|---|---| | `publication` | string | The slug, or the custom domain for publications that have moved off the Substack subdomain. Derived from the feed actually read, so it may differ from what you passed. | | `feedUrl` | string | The feed URL after redirects. The stable publication key. | | `feedTitle` | string \| null | The publication's display name. | | `postId` | string | The feed's `guid`, which for Substack equals the canonical post URL. Falls back to `link`. Use it to de-duplicate across runs. | | `title` | string \| null | Post headline, whitespace collapsed. | | `subtitle` | string \| null | The one-line subtitle. `null` when the post has none: the placeholder the feed emits is normalised away. | | `author` | string \| null | Byline. One name, even on co-authored posts. | | `publishedAt` | string \| null | Publish time as ISO 8601, converted from the feed's own format. | | `link` | string \| null | Canonical post URL. | | `contentText` | string \| null | Full body as plain text: tags stripped, entities decoded, paragraph breaks preserved as blank lines. `null` when `includeContent` is off. | | `contentHtml` | string \| null | Original markup. Non-null **only** when `includeContent` is on and `stripHtmlTags` is off. | | `isPaywalled` | boolean | `true` when the body is a subscriber preview rather than the whole post. | | `wordCount` | number | Words in the full body text, computed even when `contentText` is suppressed. `0` when the post has no body. | | `imageUrl` | string \| null | Cover image from the post's image enclosure, falling back to the first image in the body. | | `audioUrl` | string \| null | Audio enclosure. Present on podcast and voiceover posts, `null` otherwise. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, body trimmed: ```json { "publication": "newsletter.pragmaticengineer.com", "feedUrl": "https://newsletter.pragmaticengineer.com/feed", "feedTitle": "The Pragmatic Engineer", "postId": "https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne", "title": "Formal methods with Hillel Wayne", "subtitle": "Hillel Wayne explains why formal methods like TLA+ matter, how they help build reliable software …", "author": "Gergely Orosz", "publishedAt": "2026-07-29T16:22:31.000Z", "link": "https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne", "contentText": "Stream the latest episode\n\nListen and watch now on YouTube, Apple and Spotify …", "contentHtml": null, "isPaywalled": false, "wordCount": 1643, "imageUrl": "https://substackcdn.com/image/fetch/…/public/images/…jpeg", "audioUrl": "https://api.substack.com/feed/podcast/208490793/39e73f549d0e5b122bbc9dfe29a9b11a.mp3", "scrapedAt": "2026-08-06T11:47:48.866Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** The per-publication breakdown is the only place you can see what each feed contributed. ```json { "publicationsRequested": 2, "publicationsFailed": 0, "failures": [], "postsSaved": 27, "perPublication": [ { "publication": "astralcodexten", "feedUrl": "https://www.astralcodexten.com/feed", "listed": 20, "saved": 14, "paywalled": 6 } ], "filters": { "maxPostsPerPublication": 10, "sinceDate": "2026-08-01T00:00:00.000Z", "includeContent": true, "stripHtmlTags": true }, "finishedAt": "2026-08-06T11:47:49.100Z" } ``` `listed` is how many posts the feed held, `saved` how many survived `sinceDate` and the cap, and `paywalled` how many of the saved ones were previews. `publicationsRequested` counts resolvable inputs after de-duplication, so it can be lower than the length of your list without anything appearing in `failures`. `perPublication` shorter than `publicationsRequested` with no failures means two inputs resolved to the same feed after redirects and the second was skipped. `filters.sinceDate` is the normalised timestamp the run actually used, and `filters.maxPostsPerPublication` is `null` when no cap was applied. ## Behaviour to plan around - **Twenty posts, not an archive.** The feed holds only the most recent 20 and cannot be paged. `maxPostsPerPublication` above 20 changes nothing, and a publication that posts daily gives you three weeks of history at most. Schedule the Actor and accumulate forward, de-duplicating on `postId`. - **Paid posts arrive truncated.** `isPaywalled: true` means `contentText` is the free preview, so its `wordCount` is not the post's length. Filter these out before doing anything statistical with body text. - **A short post is not evidence of a paywall.** The flag is detected from how the preview ends, not from length, so trust `isPaywalled` and not `wordCount`. - **`publication` follows the redirect, not your input.** Many publications have moved to their own domain, so passing a slug can return records labelled with a domain. Key your tables on `feedUrl` or `postId`, and expect the label to change if a publication migrates between runs. - **Two inputs can silently collapse into one.** An old subdomain and the new custom domain are the same feed, which is only visible after the request, so the second is skipped rather than scraped twice. - **Unresolvable input values disappear quietly.** They are not in `failures` and not in `publicationsRequested`. Compare that count against your list length to catch typos. - **`sinceDate` is applied before the cap.** Filtering to a week and capping at 5 gives the 5 newest posts *in* that week, which is what you want for alerting. - **`contentHtml` needs two flags.** It is `null` unless `includeContent` is on and `stripHtmlTags` is off. Asking for markup while content is off yields nothing. - **One failed publication never aborts the run.** An unknown slug or a feed that is not public is final for that entry, recorded in `failures`, and the run continues. The Actor only throws when every publication failed. Transient failures are retried three times with backoff, and publications are read one per second. ## Recipes **Scheduled new-post alerting.** Cheapest possible sweep: metadata only, nothing older than the last run. ```json { "publications": ["oneusefulthing", "construction-physics", "noahpinion"], "sinceDate": "2026-08-01", "includeContent": false } ``` De-duplicate on `postId` against what you already stored, then fetch bodies only for the new ones. **Corpus building for retrieval.** Full clean text, paywalled previews still flagged so you can drop them. ```json { "publications": ["astralcodexten", "newsletter.pragmaticengineer.com"], "includeContent": true, "stripHtmlTags": true } ``` Keep rows where `isPaywalled` is false, chunk `contentText`, and carry `link`, `author` and `publishedAt` as chunk metadata. **Rendered posts with images and links intact.** ```json { "publications": ["https://www.thefp.com"], "maxPostsPerPublication": 5, "includeContent": true, "stripHtmlTags": false } ``` `contentHtml` now holds the original markup and `contentText` the flattened version, so you can render one and index the other. **Cadence comparison across competitors.** Metadata plus lengths, no bodies stored. ```json { "publications": ["noahpinion", "bariweiss.substack.com", "oneusefulthing"], "includeContent": false } ``` Group by `publication`, count posts per week from `publishedAt`, and average `wordCount` over rows where `isPaywalled` is false.