--- name: medium-articles-scraper description: Read Medium's public feeds by tag, publication or author as structured records via the Apify Actor arman-bd/medium-articles-scraper. Returns one record per article with title, byline, publish and update timestamps, canonical URL, tag list, lead image, plain-text body where the feed carries one, plus a word count and a computed reading time. Use for topic monitoring, writer or publication tracking, content-gap analysis and newsletter sourcing, especially on a schedule. Not for paywalled member-only bodies, claps, responses or follower counts, and not an archive: each feed returns only its most recent items with no pagination. --- # Medium Articles Scraper Apify Actor `arman-bd/medium-articles-scraper`. Give it tags, publication slugs, author handles, or any mix of the three, and get one dataset record per article. Every record carries the same keys whichever kind of feed it came from; only the values change. It runs without credentials. ## When to use it - Monitor one or more topics and diff on `articleId` between runs to catch new posts. - Follow a list of writers, which is the reliable way to get full article bodies. - Watch a rival publication: what it ships, when, and by whom. - Content-gap analysis across ten tags at once, counting title terms and `categories`. - Build a de-duplicated candidate list with lead images for a newsletter or digest. ## When not to use it - Member-only bodies. A paywalled post arrives as metadata plus a teaser and the Actor does not attempt to get past that. - Claps, responses, follower counts or any other engagement metric. None of it is in the feed and none of it is inferred. - Back-catalogue archives. A feed carries only its most recent items and offers no paging, so history is built by scheduling the Actor and accumulating, never in one run. - Full-text search across Medium. Filtering here happens on titles of the items a feed already returned, not across the platform. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/medium-articles-scraper').call({ tags: ['machine-learning', 'programming'], publications: ['better-programming'], authors: ['@dhh'], maxItemsPerFeed: 10, includeContent: true, searchTerms: [], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~medium-articles-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"tags":["programming"],"authors":["@dhh"],"includeContent":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/medium-articles-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `tags` | string[] | one of the three | `[]` | Tag slugs, lower-case and hyphenated: `machine-learning`, not `Machine Learning`. Take one from the address of any tag page. Tag feeds carry a teaser, never a body. | | `publications` | string[] | one of the three | `[]` | Publication slugs such as `better-programming`. A publication that has moved to its own domain also works: paste the site address or its feed address and it is used as given. Bodies are inconsistent here, see below. | | `authors` | string[] | one of the three | `[]` | Handles, with or without the leading `@`. Author feeds are the ones that reliably carry a full body. | | `maxItemsPerFeed` | integer | no | `10` | Cap per feed, applied after `searchTerms`. Medium serves at most 10 items per feed and offers no paging, so a value above 10 changes nothing there. A self-hosted publication feed may serve more. `0` means take everything the feed returned. | | `includeContent` | boolean | no | `true` | Attach `contentPlain` and the counts derived from it. Off gives a much smaller dataset of titles and metadata. | | `searchTerms` | string[] | no | `[]` | Keep an article when its title contains at least one of these, case-insensitive substring match. Applied before `maxItemsPerFeed`. Empty keeps everything. | **At least one of `tags`, `publications` or `authors` must be non-empty**, otherwise the run throws before it starts. The real decision is which of the three you use, because it decides whether you get text at all: authors give bodies, tags never do, publications sometimes do. Duplicate feeds across the three lists are collapsed and fetched once, so naming a publication and one of its authors costs two requests, not two overlapping result sets. Filtering is cheap and the caps are small, so mixing many feeds in one run is the normal shape. ## Output One record per article. Every record carries every key listed below regardless of which feed produced it. | Field | Type | Notes | |---|---|---| | `sourceType` | string | `tag`, `publication` or `author`. Tells you what to expect from `isFullContent`. | | `sourceName` | string | The slug, handle or host it came from, lower-cased. Join back to your input on this. | | `feedTitle` | string \| null | The feed's own title, for example `Better Programming - Medium`. | | `feedUrl` | string | The exact address that was fetched. Use it to tell two feeds apart when `sourceName` collides. | | `articleId` | string \| null | Medium's post ID, parsed out of the item identifier. **Null for a publication on its own domain**, which has no Medium post ID. | | `title` | string \| null | Article title. | | `author` | string \| null | The byline as the feed states it. Not a handle, so it does not round-trip into `authors`. | | `publishedAt` | string \| null | ISO 8601. | | `updatedAt` | string \| null | ISO 8601 where the feed supplies it, often absent on non-Medium feeds. | | `url` | string \| null | Canonical article URL with the feed's attribution query parameter stripped, so the same article yields the same string across feeds. This is the safest de-duplication key. | | `categories` | string[] | Every tag on the article. Often empty on publication feeds and well populated on tag feeds. | | `contentPlain` | string \| null | Body text with markup stripped and entities decoded, trailing feed furniture removed. A one-line teaser when the feed carried no body. **Null when `includeContent` is false.** | | `isFullContent` | boolean | `true` only when the feed actually supplied a body. Independent of `includeContent`: it describes the feed, not your request. | | `wordCount` | number \| null | Words in `contentPlain`. Null unless `isFullContent` is true **and** `includeContent` was on. | | `readingTimeMinutes` | number \| null | Computed as the word count divided by 200, rounded up, minimum 1. Not Medium's own badge. Null whenever `wordCount` is. | | `imageUrl` | string \| null | First real image in the article. The view-tracking pixel is filtered out rather than returned. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A record from a publication feed that carried a body, `contentPlain` trimmed: ```json { "sourceType": "publication", "sourceName": "better-programming", "feedTitle": "Better Programming - Medium", "feedUrl": "https://…", "articleId": "bf37baef8f27", "title": "Let a thousand programming publications bloom.", "author": "Tony Stubblebine", "publishedAt": "2023-11-10T18:18:10.000Z", "updatedAt": "2023-11-27T17:53:37.557Z", "url": "https://medium.com/better-programming/let-a-thousand-programming-publications-bloom-bf37baef8f27", "categories": [], "contentPlain": "I'm putting Better Programming on hiatus to make room for other programming publications. …", "isFullContent": true, "wordCount": 1305, "readingTimeMinutes": 7, "imageUrl": "https://cdn-images-1.medium.com/max/1024/1*tKikPWjE4MZ5WgdfZuyvEg.png", "scrapedAt": "2026-08-06T11:58:36.656Z" } ``` The same shape from a tag feed, teaser instead of body: ```json { "sourceType": "tag", "sourceName": "programming", "title": "6 Best AI Tools for Software Engineers in 2026 (Compared)", "author": "Shahid Bahadur", "url": "https://shahidyousafxai.medium.com/6-best-ai-tools-for-software-engineers-in-2026-compared-ddb8992886d2", "categories": ["web-development", "programming", "artificial-intelligence", "productivity", "software-engineering"], "contentPlain": "AI has become a daily part of software development. Whether you're debugging code, refacto", "isFullContent": false, "wordCount": null, "readingTimeMinutes": null, "imageUrl": "https://cdn-images-1.medium.com/max/1536/1*gGldf-aWBPpS11vTV4p9Pg.png" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "feedsRequested": ["tag:programming", "publication:better-programming", "author:dhh"], "feedsFailed": 1, "failures": [ { "feed": "nosuchhandle", "type": "author", "url": "https://…", "error": "feed not found (404) …" } ], "articlesSaved": 24, "filters": { "searchTerms": ["rust"], "maxItemsPerFeed": 10, "includeContent": true }, "finishedAt": "2026-08-06T11:58:40.112Z" } ``` `feedsRequested` is an **array of `type:name` strings, not a count**; take its length for the number of feeds after de-duplication. `feedsFailed` counts feeds that could not be read at all. A feed that answered with zero items is not a failure and does not appear in `failures`, so `articlesSaved` short of your expectation with an empty `failures` means either a tag with nothing recent or a `searchTerms` list that matched nothing. ## Behaviour to plan around - **Ten items per feed, no paging.** Medium's feeds serve their most recent items only and there is no offset, page or limit parameter to reach further back. Raising `maxItemsPerFeed` above 10 has no effect on Medium-hosted feeds. Archives are built by scheduling the run and accumulating on `url` or `articleId`. - **`isFullContent` is the field to branch on, not `sourceType`.** Tag feeds are always false and author feeds are almost always true, but publication feeds are genuinely mixed: one item in a feed can carry a body while the other nine do not. Nothing is fabricated to hide the gap. - **`wordCount` and `readingTimeMinutes` are null far more often than they are set.** They require both a real body in the feed and `includeContent: true`. Reading time is derived at 200 words per minute, not taken from Medium's own badge. - **A non-existent tag is not an error.** It answers normally with zero items, so it never appears in `failures`. A bad author handle or publication slug is a not-found and does appear there. - **`articleId` is null for publications on their own domain.** Those left Medium and now publish through something else. Their feeds still parse, but the field mix differs and `url` is the only reliable key. - **`url` is canonicalised.** The attribution query parameter the feed appends is removed, so the same article discovered through a tag feed and an author feed produces one identical string. De-duplicate on it rather than on `title`. - **Filtering happens before the cap.** `searchTerms` narrows first, then `maxItemsPerFeed` truncates what is left, so a cap of 3 with a filter gives you the three most recent *matching* items rather than matches among the three most recent. - **One bad feed never aborts the run.** It lands in `RUN_SUMMARY.failures` and the rest continue. The Actor only throws when every feed failed. - **Transient failures are retried** three times with linear backoff. Not-found and non-feed responses are final and are not retried. Requests are spaced about 400 ms apart. ## Recipes **Hourly topic monitor.** Metadata only, several tags, cheap enough to run often. ```json { "tags": ["machine-learning", "programming", "javascript", "devops"], "includeContent": false } ``` Upsert on `url` and treat anything new as a fresh post; `publishedAt` gives the ordering. **Follow writers for full text.** Author feeds are the ones that carry bodies. ```json { "authors": ["@dhh", "shahidyousafxai"], "includeContent": true, "maxItemsPerFeed": 10 } ``` Filter the result to `isFullContent: true` before feeding anything into a summariser. **Keyword watch across a mixed set.** Titles filtered, then capped. ```json { "tags": ["machine-learning", "data-science"], "publications": ["better-programming"], "authors": ["@dhh"], "searchTerms": ["llm", "rust", "kubernetes"], "maxItemsPerFeed": 10, "includeContent": true } ``` An empty result here usually means the terms, not the feeds: check `articlesSaved` against `feedsFailed` in the summary. **Publication watch.** One publication, everything it published recently. ```json { "publications": ["better-programming"], "maxItemsPerFeed": 0, "includeContent": true } ``` Group by `author` and count to see who is carrying the publication.