--- name: lemmy-posts-scraper description: Read posts from any Lemmy instance via the Apify Actor arman-bd/lemmy-posts-scraper. Returns one record per post with title, self-text body, outbound link, score with the upvote and downvote split, comment count, community and author details, the federation trail showing where each post and author originated, and optionally the top-scoring top-level comments. Use when a task needs fediverse discussion tracking, brand monitoring outside the mainstream aggregators, newsletter curation or controversy scoring. Not for Mastodon, Reddit, private communities or full comment trees. --- # Lemmy Scraper: Federated Communities & Posts Apify Actor `arman-bd/lemmy-posts-scraper`. Name one or more Lemmy instances, optionally narrow to communities, and get one dataset record per post. Because Lemmy federates, each record carries both the instance you queried and the instance the content actually came from. It takes no credentials. ## When to use it - Tracking technology or product discussion across several communities in one run. - Brand monitoring on a schedule, sweeping a set of instances and searching the bodies you collect. - Newsletter curation: a `TopWeek` sort over a handful of communities gives a ranked shortlist. - Controversy scoring, which needs `upvotes` and `downvotes` separately rather than a net score. - Federation research, comparing `communityInstance` against `creatorInstance` to see how content crosses instances. ## When not to use it - Other fediverse software. Mastodon, Reddit and similar aggregators are not covered. - Full comment trees. Only top-level comments are fetched, and only the highest scoring of them. - Private or login-gated communities. There is no credential input. - Per-user history. You get the author fields attached to a post, not a profile or a user's post list. - Full-text search. The Actor reads sorted feeds; filter the bodies yourself afterwards. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/lemmy-posts-scraper').call({ instances: ['lemmy.world', 'programming.dev'], communities: ['technology'], sortBy: 'TopWeek', maxPostsPerInstance: 25, includeComments: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~lemmy-posts-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"instances":["lemmy.world"],"communities":["technology"],"sortBy":"TopWeek","maxPostsPerInstance":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/lemmy-posts-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `instances` | string[] | no | `["lemmy.world"]` | Hostnames or URLs. A scheme and any path are stripped, the host is lower-cased, duplicates dropped. Each instance is queried independently. An entry that is not a valid host is discarded; if none survive, the run throws. | | `communities` | string[] | no | `[]` | Applied to **every** instance. Accepts `technology`, `!technology@lemmy.world` or a community URL: the leading `!`, the `@domain` suffix and the URL wrapper are all stripped to the bare slug. Empty reads each instance's whole front page. | | `sortBy` | string | no | `"Hot"` | One of `Hot`, `Active`, `New`, `Scaled`, `TopDay`, `TopWeek`, `TopMonth`, `TopYear`, `TopAll`, `MostComments`, `NewComments`, `Controversial`. An unrecognised value throws before any request. | | `maxPostsPerInstance` | integer | no | `50` | Total per instance, split evenly across the communities listed. Minimum `1`. | | `includeComments` | boolean | no | `false` | Fetch top-level comments. One extra request per post that has any, so a 500-post run becomes roughly 510 requests. | | `maxCommentsPerPost` | integer | no | `5` | Comments kept per post, highest score first. Minimum `1`. Ignored when `includeComments` is off. | **`communities` divides the budget, it does not multiply it.** `maxPostsPerInstance: 50` with four communities is about 13 posts each, not 50 each, so raise the cap in proportion when you widen the community list. The other real decision is `includeComments`: off, a run costs one request per 50 posts and is fast enough to schedule tightly; on, it costs an extra request for every post that has any comments at all, which dominates the run. Turn it on when the discussion is the point, off when the post metrics are. ## Output One record per post. | Field | Type | Notes | |---|---|---| | `instance` | string | The instance you queried, not necessarily where the post was written. | | `postId` | number | The post id **local to `instance`**. The same post read from another instance has a different id. | | `title` | string \| null | Post title. | | `body` | string \| null | Self-text, Markdown as authored. `null` or empty on link posts. | | `url` | string \| null | Outbound link for link posts. | | `postUrl` | string | Canonical link on the instance you queried, built from `instance` and `postId`. | | `apId` | string \| null | ActivityPub id: the post's true origin URL across the fediverse. **This is the cross-instance identity.** | | `communityName` | string \| null | Community slug. | | `communityTitle` | string \| null | Community display name. | | `communityUrl` | string \| null | Community actor URL. | | `communityInstance` | string \| null | Host parsed out of `communityUrl`. | | `creatorName` | string \| null | Author handle. | | `creatorDisplayName` | string \| null | Author display name, often the same as the handle. | | `creatorUrl` | string \| null | Author actor URL. | | `creatorInstance` | string \| null | Host parsed out of `creatorUrl`, the author's home instance. | | `creatorIsBot` | boolean | Bot flag as the author's instance declares it. Defaults to `false`. | | `score` | number \| null | Net score. | | `upvotes` | number \| null | Reported separately, so controversy is computable. | | `downvotes` | number \| null | Same. | | `commentCount` | number \| null | Comment count as the queried instance sees it. | | `published` | string \| null | ISO 8601 with microseconds. | | `updated` | string \| null | Last edit, `null` when never edited. | | `newestCommentAt` | string \| null | Time of the most recent comment. | | `featured` | boolean | True when pinned in the community **or** site-wide. The two are collapsed into one flag. | | `locked` | boolean | Locked to new comments. | | `nsfw` | boolean | NSFW flag. | | `thumbnailUrl` | string \| null | Link preview image. | | `embedTitle` | string \| null | Link preview title. | | `embedDescription` | string \| null | Link preview description. | | `comments` | object[] \| null | `null` unless `includeComments` is on and the post has at least one comment. Keys: `commentId`, `body`, `creatorName`, `score`, `upvotes`, `downvotes`, `published`, `url`. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "instance": "lemmy.world", "postId": 50183504, "title": "New EU law requires devices to be repairable", "body": "The products covered by the obligation to repair currently include: washing machines …", "url": "https://www.nporadio1.nl/fragmenten/nos-met-het-oog-op-morgen", "postUrl": "https://lemmy.world/post/50183504", "apId": "https://piefed.zip/c/technology@lemmy.world/p/1699468/new-eu-law-requires-devices-to-be-repairable", "communityName": "technology", "communityTitle": "Technology", "communityUrl": "https://lemmy.world/c/technology", "communityInstance": "lemmy.world", "creatorName": "inari", "creatorDisplayName": "inari", "creatorUrl": "https://piefed.zip/u/inari", "creatorInstance": "piefed.zip", "creatorIsBot": false, "score": 1764, "upvotes": 1768, "downvotes": 4, "commentCount": 283, "published": "2026-08-01T20:40:59.572119Z", "updated": "2026-08-01T20:42:20.713133Z", "newestCommentAt": "2026-08-06T10:29:32.713285Z", "featured": false, "locked": false, "nsfw": false, "thumbnailUrl": "https://radioimages.npox.nl/RAD1/image.webp", "embedTitle": "Apparaten verplicht repareerbaar door nieuwe EU-wet", "embedDescription": "Iedere wasmachine, telefoon of televisie die je vanaf morgen in Europa koopt …", "comments": [ { "commentId": 25077535, "body": "Hopefully this will benefit other countries indirectly …", "creatorName": "dan1101", "score": 199, "upvotes": 199, "downvotes": 0, "published": "2026-08-01T21:01:30.164757Z", "url": "https://lemmy.world/comment/25077535" } ], "scrapedAt": "2026-08-06T11:42:30.835Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "instancesRequested": 3, "targetsFailed": 1, "failures": [ { "target": "lemmy.ml/c/technology", "error": "not found (404), check the instance or community name" } ], "postsSaved": 87, "filters": { "instances": ["lemmy.world", "programming.dev", "lemmy.ml"], "communities": ["technology"], "sortBy": "TopWeek", "maxPostsPerInstance": 50, "includeComments": false, "maxCommentsPerPost": 5 }, "finishedAt": "2026-08-06T11:42:31.002Z" } ``` A target is one instance and community pair, so `targetsFailed` counts pairs rather than instances: three instances times one community is three targets. Each `failures` entry names the pair in `target`, and a not-found there means that community is not carried on that instance, which is normal in a federated network. `postsSaved` below `instancesRequested * maxPostsPerInstance` is expected whenever a feed was shorter than the budget. There is no per-target saved count, so if you need one, group the dataset by `instance` and `communityName` yourself. ## Behaviour to plan around - **The same post appears once per instance you query.** Read one community from three instances and a federated post comes back three times with three different `postId` values. De-duplicate on `apId`, which is stable across the fediverse. `postId` alone will not do it. - **`comments: null` is ambiguous by design.** It means either `includeComments` was off, or it was on and the post has no comments. Check `commentCount` to tell them apart. Comments are never an empty array. - **Only top-level comments, only the top few.** Replies are not fetched at any depth, and `maxCommentsPerPost` is applied after the response arrives, so the request cost is the same whether you keep one comment or twenty. - **The per-instance budget splits across communities and rounds up.** Four communities and a cap of 50 gives 13 each, so the run can save slightly more than the cap. - **`featured` collapses two flags.** Pinned within the community and pinned site-wide both report `true`, and the record cannot distinguish them. - **`body` is empty on link posts.** That is not a scrape failure. For those posts the content lives in `url`, `embedTitle` and `embedDescription`. - **Stick to the sort values above.** The Actor recognises a few extra time windows internally, but the input schema does not advertise them and the platform validates input against the schema, so an undocumented value can be rejected before the run starts. - **A missing community never aborts the run.** It is recorded in `RUN_SUMMARY.failures` and the other targets continue. The Actor throws only when every target failed. - **Not-found is final, transient failures are retried** three times with linear backoff. Re-running a wrong community name will fail identically. - **Vote counts are the queried instance's view.** Federation lag means two instances can report slightly different `score` values for the same post at the same moment. ## Recipes **Track a topic across instances.** One community, three instances, ranked by the week. ```json { "instances": ["lemmy.world", "programming.dev", "lemmy.ml"], "communities": ["technology"], "sortBy": "TopWeek", "maxPostsPerInstance": 100 } ``` De-duplicate on `apId`, then sort by `score`. **Discussion mining with replies.** Fewer posts, comments on, sorted by activity. ```json { "instances": ["lemmy.world"], "communities": ["technology", "asklemmy"], "sortBy": "MostComments", "maxPostsPerInstance": 40, "includeComments": true, "maxCommentsPerPost": 10 } ``` Concatenate `title`, `body` and `comments[].body` for classification or summarisation. **Controversy scan.** The sort surfaces them; the vote split quantifies them. ```json { "instances": ["lemmy.world"], "sortBy": "Controversial", "maxPostsPerInstance": 100 } ``` Rank by `downvotes / (upvotes + downvotes)` and keep `locked` as a second signal. **Scheduled new-post feed.** Strictly chronological, cheap, diffable. ```json { "instances": ["programming.dev"], "communities": ["programming"], "sortBy": "New", "maxPostsPerInstance": 50 } ``` Diff on `apId` against the previous run to get only what appeared since.