--- name: devto-articles-scraper description: Read published DEV.to articles as structured records via the Apify Actor arman-bd/devto-articles-scraper. Returns title, summary, tag list, public reaction and comment counts, reading time, publish timestamp, canonical URL, cover image, author handle, display name and GitHub username, plus the full markdown body on request. Use for developer-content trend tracking, tag-level engagement analysis, author performance monitoring, technical-writer sourcing and cross-post attribution. Not for comment bodies, private or unpublished drafts, follower counts, reading lists or any account-scoped DEV.to data. --- # DEV.to Scraper: Articles, Tags & Author Metrics Apify Actor `arman-bd/devto-articles-scraper`. Give it tags, an author handle, or nothing at all, and get one dataset record per published article. It runs without credentials. ## When to use it - Sweeping a set of tags for the week's or month's best-performing posts. - Tracking one author's whole back catalogue and diffing engagement over time. - Ranking technical writers in a niche by reactions, with their GitHub handle attached. - Building a corpus of article text on a topic, with `includeBody` on. - Daily change detection on a tag feed, diffing on `id` to isolate new posts. ## When not to use it - Comment text, follower counts, reading lists, or anything behind a DEV.to login. Only `commentsCount` is available, never the comments themselves. - Drafts, scheduled posts or unpublished articles. The feed carries published posts only. - Full-text search across DEV.to. There is no free-text query input; you filter by tag and by author, then filter the dataset yourself. - Organisation or podcast content. The Actor reads the article feed. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/devto-articles-scraper').call({ tags: ['python', 'webdev'], topArticles: 7, includeBody: false, maxResults: 100, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~devto-articles-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"tags":["rust"],"topArticles":30,"includeBody":false,"maxResults":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/devto-articles-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `tags` | string[] | no | `[]` | DEV.to tag names, one query each. Tags are lowercase and unspaced upstream, so `machinelearning`, not `Machine Learning`. A leading `#`, inner spaces and mixed case are normalised away. A tag that does not exist returns zero articles rather than an error. | | `username` | string | no | `""` | A single author handle, for example `ben`. A full profile URL or a leading `@` is accepted and reduced to the handle. Runs as its own query. | | `topArticles` | integer | no | `0` | Order every query by reactions within the last N days instead of by recency. `7` is the week's best, `30` the month's. `0` keeps the chronological feed. Maximum 3650. | | `includeBody` | boolean | no | `false` | Also fetch the complete markdown source of each article. Costs one extra request per article plus a short pause, so it roughly doubles run time. | | `maxResults` | integer | no | `100` | Total cap across every source combined. `0` means keep paging until all sources are exhausted. Maximum 100000. | **`tags` and `username` are a union, never an intersection.** The source ignores the author filter when a tag is also present, so the Actor issues each tag and the username as separate queries and merges them, de-duplicating on `id`. Passing `tags: ["rust"]` with `username: "ben"` gives you the whole Rust feed plus everything Ben wrote, not Ben's Rust posts. For the intersection, pass only the username and filter the dataset on `tags` yourself. `maxResults` is consumed in source order, tags first and the username query last, so a tight cap can leave the author query with nothing: put the source you care most about first, or raise the cap. ## Output One record per article saved. | Field | Type | Notes | |---|---|---| | `id` | number | DEV.to article ID. The de-duplication key within a run and the join key across runs. | | `title` | string | Headline. | | `description` | string \| null | The short summary line, usually auto-generated from the opening text. | | `url` | string | The article's page on DEV.to. | | `canonicalUrl` | string \| null | The author's declared canonical URL. Frequently a different domain, because the post was cross-posted from their own blog. | | `tags` | string[] | Every tag on the post. `[]` when untagged. | | `reactionsCount` | number | Public reactions. Falls back to the positive-reaction count, and to `0` if neither is present. | | `commentsCount` | number | Comment count only. `0` when absent. | | `readingTimeMinutes` | number \| null | DEV.to's own estimate, in whole minutes. | | `publishedAt` | string \| null | Publish timestamp, ISO 8601. | | `author` | string \| null | Author handle, the first path segment of their profile URL. | | `authorName` | string \| null | Author display name. | | `authorGithub` | string \| null | GitHub username, only where the author linked one. Commonly `null`. | | `coverImage` | string \| null | Declared cover image, falling back to the generated share card, so it is rarely `null` even on posts with no real cover. | | `bodyMarkdown` | string \| null | Full article source in markdown. Present only when `includeBody` is on; the key is absent from the record entirely when it is off, and `null` when that one body could not be fetched. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "id": 4323385, "title": "General Challenge Updates Moving Forward", "description": "Hey all, as part of our expanding challenge program we want to update and clarify some challenge.", "url": "https://dev.to/thepracticaldev/general-challenge-updates-moving-forward-5h39", "canonicalUrl": "https://dev.to/thepracticaldev/general-challenge-updates-moving-forward-5h39", "tags": ["devchallenge"], "reactionsCount": 14, "commentsCount": 1, "readingTimeMinutes": 2, "publishedAt": "2026-08-05T19:27:14Z", "author": "thepracticaldev", "authorName": "dev.to staff", "authorGithub": null, "coverImage": "https://media2.dev.to/dynamic/image/width=1200,height=627./mgecj41w91psrct0upxp.png", "bodyMarkdown": "---\ntitle: General Challenge Updates Moving Forward\n---\n\nHey all …", "scrapedAt": "2026-08-06T12:00:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "sourcesRequested": 2, "sourcesFailed": 0, "failures": [], "articlesSaved": 100, "bodiesFetched": 100, "bodiesFailed": 0, "filters": { "tags": ["python", "webdev"], "username": null, "topArticles": 7, "includeBody": true, "maxResults": 100 }, "finishedAt": "2026-08-06T12:00:04.812Z" } ``` Three things to check. A non-empty `failures` array means a whole source query aborted; each entry carries `source` (the `#tag` or `@handle` label) and `error`. `bodiesFailed` above zero means some records have `bodyMarkdown: null` for a fetch reason rather than because the post is empty. And `articlesSaved` sitting exactly on `maxResults` means the cap truncated, so later sources may never have run at all. When every source fails the Actor throws rather than finishing, so a completed run always saved something. ## Behaviour to plan around - **`bodyMarkdown` is absent, not null, when `includeBody` is off.** The key is omitted from the record. Code that tests `record.bodyMarkdown === null` to mean "no body" will be wrong; test for `undefined` too. - **`includeBody` costs one extra request per article.** On a 300-article run that is 300 extra round trips plus a pause between each. Leave it off for metrics-only sweeps and change detection. - **De-duplication is global across sources.** An article carrying two of your tags is saved once, under whichever source reached it first. Per-source counts in the run log therefore do not sum to `articlesSaved`. - **`topArticles` applies to every query in the run**, not to one of them. There is no way to take one tag chronologically and another by reactions in a single run. - **`maxResults: 0` means unlimited.** On a broad tag with no cap this pages until the feed is exhausted, which can be a very long run. Set a cap unless you mean it. - **`canonicalUrl` frequently points off DEV.to.** That is correct data, not a bug, and it is the right URL for attribution. It also matches `url` on posts written natively. - **`coverImage` is almost never null**, because a generated share card stands in when the author set no cover. Do not treat its presence as evidence of a real cover image. - **Transient failures are retried** three times with linear backoff. A not-found is final immediately, which is what a mistyped username produces. - **`reactionsCount` is public reactions in aggregate**, not likes. There is no breakdown by reaction type here. ## Recipes **Weekly trending sweep across your tags.** Reactions-ranked, last seven days, bodies off so it stays quick. ```json { "tags": ["ai", "webdev", "python"], "topArticles": 7, "includeBody": false, "maxResults": 300 } ``` Sort the dataset by `reactionsCount` and group by `tags` to see which topic is moving. **One author's full back catalogue.** No cap, so it pages to the end of their history. ```json { "username": "ben", "topArticles": 0, "includeBody": false, "maxResults": 0 } ``` Re-run on a schedule and diff `reactionsCount` and `commentsCount` keyed on `id`. **Writer shortlist in a niche.** Ninety days of the tag's best, with text for judging quality rather than only counts. ```json { "tags": ["rust"], "topArticles": 90, "includeBody": true, "maxResults": 200 } ``` Rank on `reactionsCount`, keep records where `authorGithub` is non-null, group by `author`. **Daily new-post watch.** Chronological feed, small cap, cheap. ```json { "tags": ["kubernetes"], "topArticles": 0, "includeBody": false, "maxResults": 100 } ``` Keep the set of `id` values from the previous run and alert on anything new.