--- name: discourse-forum-scraper description: Export topics from any Discourse-powered community via the Apify Actor arman-bd/discourse-forum-scraper. Returns one record per topic with title, category, tags, views, reply and like counts, timestamps, the list of people who posted, the excerpt, and optionally every reply body as clean plain text. Use when a task needs community mining, feature-request ranking, support corpora for retrieval, or a scheduled watch on new threads across several forums. Not for private categories, direct messages, user profiles or non-Discourse forum software. --- # Discourse Forum Scraper: Topics, Posts & Users Apify Actor `arman-bd/discourse-forum-scraper`. Give it forum addresses, get one dataset record per topic. Every host is confirmed to be a Discourse install before it is crawled, so an unusable input is reported as such rather than returning zero rows. Post bodies are optional and come back as readable text, not markup. ## When to use it - Mining a product community for feature requests, ranked by `likeCount` or `views`. - Building a retrieval corpus out of a support category, with every reply in plain text. - Watching several communities on a schedule and diffing on `topicId` for new threads. - Measuring engagement across forums, where `views`, `replyCount` and `likeCount` need to sit in one table. - Pulling a specific category out of a large forum without paging its global feed. ## When not to use it - Anything requiring an account: private categories, personal messages, unlisted topics. There is no credential input. - User profiles, badges, trust levels or follower graphs. You get the usernames who posted on a topic and nothing else about them. - Forums that are not Discourse. Other software is detected and skipped, not parsed. - Full-text search across a forum. The Actor walks list feeds; it does not query the forum's search. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/discourse-forum-scraper').call({ forums: ['forum.obsidian.md'], categories: ['feature-requests'], sortBy: 'top', includePosts: false, maxTopicsPerForum: 200, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~discourse-forum-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"forums":["meta.discourse.org","discuss.python.org"],"sortBy":"top","maxTopicsPerForum":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/discourse-forum-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `forums` | string[] | yes | | Forum addresses. Scheme, `www.` and any trailing path are stripped, the result is lower-cased, and duplicates are dropped. `meta.discourse.org` and a full topic URL on that host collapse to the same entry. An entry that is not a valid host is discarded silently; if nothing survives, the run throws. | | `categories` | string[] | no | `[]` | Category slugs, lower-cased. The slug is the short text segment in a category's URL, not its display name. Sub-categories match too. Empty means the whole forum. | | `sortBy` | string | no | `"latest"` | `latest` (bumped by activity), `top` (most engaged, all time), `new` (ordered by creation date). An unrecognised value falls back to `latest` without complaint. | | `includePosts` | boolean | no | `false` | Fetch every reply body. Costs at least one extra request per topic, more on long threads. | | `maxTopicsPerForum` | integer | no | `100` | Cap per forum, shared across that forum's categories. `0` means no cap. Maximum `100000`. | **`includePosts` is the one setting that changes the shape of the run.** Off, a forum costs a handful of list requests and you still get `postsCount`, `replyCount`, `posters` and `excerpt`, which is enough for ranking, monitoring and dashboards. On, every single topic costs its own request or several, so a 500-topic run becomes minutes rather than seconds. Turn it on to build a corpus, off to watch. `categories` is the other lever worth understanding: it changes which route is crawled rather than filtering afterwards, so pulling a quiet category out of a busy forum is cheap, not expensive. ## Output One record per topic. | Field | Type | Notes | |---|---|---| | `forum` | string | Normalised host the topic came from. Join your input list on this, not on what you typed. | | `topicId` | number | Topic id, unique within a forum but not across forums. Key on `forum` plus `topicId`. | | `title` | string | Topic title. | | `slug` | string | URL slug. | | `url` | string | Direct link to the topic, rebuilt from `forum`, `slug` and `topicId`. | | `categoryId` | number \| null | Category id as the forum reports it. | | `categoryName` | string \| null | Resolved display name, sub-categories included. `null` when the name lookup failed for that forum. | | `tags` | string[] | Topic tags. Empty array when the forum has tagging off. | | `postsCount` | number | Total posts including the opening one. | | `replyCount` | number | Replies excluding the opening post. Not always `postsCount - 1`. | | `views` | number | View count as the forum reports it. | | `likeCount` | number | Likes across the topic. | | `createdAt` | string \| null | ISO 8601, when the topic was opened. | | `lastPostedAt` | string \| null | Last post time, falling back to the last bump. | | `posters` | object[] | Everyone the forum lists on the topic. Keys: `username`, `name`, `role`. `role` is free text such as `Original Poster, Most Recent Poster`. | | `excerpt` | string | The forum's own summary of the opening post, HTML stripped to plain text. | | `posts` | object[] \| null | **Key is absent entirely unless `includePosts` is on.** `null` when it was on but that topic's posts could not be read. Otherwise every post in thread order, keys `postId`, `postNumber`, `username`, `name`, `createdAt`, `updatedAt`, `replyCount`, `reads`, `score`, `text`. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "forum": "meta.discourse.org", "topicId": 1, "title": "New to Discourse? Start here", "slug": "new-to-discourse-start-here", "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1", "categoryId": 67, "categoryName": "Announcements", "tags": [], "postsCount": 1, "replyCount": 5, "views": 117628, "likeCount": 435, "createdAt": "2013-02-01T04:52:28.073Z", "lastPostedAt": "2013-02-01T04:52:28.397Z", "posters": [ { "username": "Discourse", "name": null, "role": "Original Poster, Most Recent Poster" } ], "excerpt": "This is our official community, where you can connect, ask questions, and share ideas …", "posts": [ { "postId": 2019829, "postNumber": 1, "username": "Discourse", "name": null, "createdAt": "2013-02-01T04:52:28.073Z", "updatedAt": "2025-09-23T19:13:50.543Z", "replyCount": 0, "reads": 16, "score": 23, "text": "This is our official community …" } ], "scrapedAt": "2026-08-06T11:39:37.084Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "forumsRequested": 4, "forumsFailed": 2, "failures": [ { "forum": "example.com", "kind": "forum", "error": "not found (404)" }, { "forum": "meta.discourse.org/c/support", "kind": "category", "error": "category \"support\" not found" } ], "topicsSaved": 80, "postsSaved": 0, "filters": { "forums": ["meta.discourse.org", "discuss.python.org", "example.com"], "categories": [], "sortBy": "latest", "includePosts": false, "maxTopicsPerForum": 40 }, "finishedAt": "2026-08-06T11:39:39.384Z" } ``` `kind` is the field that tells you what to do next. `forum` means the host never got as far as being crawled, either because it could not be reached or because it is not a Discourse install: check the address. `category` means the forum was fine but that slug does not exist there, so the topics you expected are simply absent. `list` means a crawl broke partway, so the rows for that route are incomplete rather than missing. Note that for `category` and `list` the `forum` field carries a route label, host plus category path, not a bare host. `postsSaved` stays at `0` whenever `includePosts` was off, which is not an error. ## Behaviour to plan around - **`posts` is absent, not empty, when `includePosts` is off.** Code that reads `record.posts.length` throws rather than seeing `0`. Check for the key. - **`posts: null` means something different from `posts: []`.** `null` is a topic whose bodies could not be read while the rest of the run succeeded; the topic row is still saved. An empty array would mean a topic with no readable posts. - **Long threads are fetched completely.** The full ordering is walked, not just the first chunk a topic page embeds, so `posts` for a 300-reply thread contains 300 posts and costs proportionally more requests. - **`maxTopicsPerForum` is shared across categories.** With three categories and a cap of 100, the first category can consume the whole allowance. Order the list by priority, or run one category per call if you need a guaranteed split. - **An ambiguous category slug crawls every match.** The same slug can exist under several parents; all of them are crawled and their topics land in the same dataset. Use `categoryId` to separate them afterwards. - **`sortBy: "new"` is the latest feed re-ordered by creation date**, which is what an anonymous visitor sees. `sortBy: "top"` is explicitly all-time, not the forum's default window. - **`categoryName` can be null while `categoryId` is set.** The name lookup runs once per forum and the crawl continues without it. Resolve names later rather than treating the row as broken. - **Not every Discourse host is publicly readable.** Ones that are not are recorded in `failures` with `kind: "forum"` and skipped; no input changes that outcome. - **One bad forum never aborts the run.** The Actor throws only when every forum failed. Transient failures are retried three times with backoff; not-found, access-denied and wrong-content-type failures are final. - **Requests are paced**, so a large crawl with `includePosts` on should be treated as a scheduled job rather than something to await inside a request handler. ## Recipes **Rank feature requests.** One category, most engaged first, no bodies. ```json { "forums": ["forum.obsidian.md"], "categories": ["feature-requests"], "sortBy": "top", "includePosts": false, "maxTopicsPerForum": 500 } ``` Sort the rows by `likeCount`, then `views`, and read `excerpt` for the ask. **Build a support knowledge base.** Full bodies, most engaged threads, one forum. ```json { "forums": ["meta.discourse.org"], "categories": ["support"], "sortBy": "top", "includePosts": true, "maxTopicsPerForum": 1000 } ``` Chunk `posts[].text` for embedding and keep `url` as the citation. **Daily new-thread watch across communities.** Small cap, newest first, cheap. ```json { "forums": ["discuss.python.org", "meta.discourse.org", "community.openai.com"], "sortBy": "new", "includePosts": false, "maxTopicsPerForum": 50 } ``` Diff against the previous run on `forum` plus `topicId`, then re-run with `includePosts: true` only on the ids that are new. **Sentiment or theme classification.** Newest threads with bodies, capped so the run stays bounded. ```json { "forums": ["discuss.python.org"], "sortBy": "new", "includePosts": true, "maxTopicsPerForum": 100 } ``` Feed `title` plus the concatenated `posts[].text` to whatever classifier you already run, and keep `createdAt` for the time axis.