---
name: lobsters-stories-scraper
description: Read Lobsters stories as structured records via the Apify Actor arman-bd/lobsters-stories-scraper. Returns one record per story with title, outbound URL, score, flag count, comment count, submission time, submitter, the full tag list and the comments permalink, from the front page, the newest queue or any tag feed, with optional tag and minimum-score filtering. Use for engineering digests, niche topic tracking, release monitoring, newsletter sourcing or joining against Hacker News on the outbound URL. Not for comment threads, user profiles, article full text, historical date ranges or voting data.
---

# Lobsters Stories Scraper

Apify Actor `arman-bd/lobsters-stories-scraper`. Name the feeds you want, get one
dataset record per story. Lobsters is invite-only and much smaller than the general
technical aggregators, so the volume is low and the signal per story is high. It runs
without credentials, and with no input at all it returns the current front page.

## When to use it

- A daily engineering digest: the front page above a score threshold, straight into
  Slack or an email.
- Watching a narrow tag (`t/rust`, `t/nix`, `t/compilers`) for anything that clears a
  score bar.
- Release monitoring: the `release` tag is a clean stream of version announcements.
- Newsletter sourcing, where you want a scored, tagged, deduplicated candidate list
  rather than a page to read.
- Cross-community comparison: join on `url` against another aggregator to see which
  audiences picked a story up.

## When not to use it

- Comment threads. Counts and a permalink are returned; the comments themselves are
  not.
- The linked articles. `url` is handed to you so you can fetch them separately.
- User profiles, karma, invite trees or voting detail.
- A date-ranged historical pull. There is no date parameter: you page the newest queue
  and stop on `createdAt` yourself.

## Call it

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('arman-bd/lobsters-stories-scraper').call({
  feeds: ['hottest', 't/rust'],
  tags: [],
  minScore: 5,
  maxStories: 60,
  maxPagesPerFeed: 4,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const { value: summary } = await client
  .keyValueStore(run.defaultKeyValueStoreId)
  .getRecord('RUN_SUMMARY');
```

One-shot over HTTP, when you want the rows back in the same request:

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~lobsters-stories-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"feeds":["hottest"],"minScore":10,"maxStories":50}'
```

The Actor is also exposed through Apify's MCP server as
`arman-bd/lobsters-stories-scraper`, so an MCP-capable agent can call it with no extra
wiring.

## Input

| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
| `feeds` | string[] | no | `["hottest"]` | `hottest` for the front page, `newest` for submission order, or a tag feed written as `rust`, `t/rust`, or `rust,go` for one combined feed carrying any of those tags. A full site URL is accepted and normalised, as is a trailing page number, which is discarded. Case is folded, duplicates removed. An empty list means `hottest`. |
| `tags` | string[] | no | `[]` | Keep only stories carrying at least one of these tags, exact match, case-insensitive. Applied after fetching, so it works on `hottest` and `newest` as well as on tag feeds. |
| `minScore` | integer | no | `0` | Drop stories scoring below this. Scores here are small: 10 is already a good story, 50 is a hit. Negative values are accepted. |
| `maxStories` | integer | no | `100` | Total saved across all feeds, not per feed. `0` means no limit, in which case `maxPagesPerFeed` is the only bound. |
| `maxPagesPerFeed` | integer | no | `4` | How deep to page each feed. Every page is 25 stories, so 4 pages is 100 stories per feed. Clamped to 1 to 100. |

**`maxStories` and `maxPagesPerFeed` bound the run from opposite ends and the tighter
one wins.** `maxPagesPerFeed` limits what is fetched; `maxStories` limits what is kept
after `tags` and `minScore` have thrown rows away. A strict filter over a quiet tag
therefore returns almost nothing at the default depth, and the fix is to raise
`maxPagesPerFeed`, not `maxStories`. In the other direction, feeds are consumed in the
order you list them, so a low `maxStories` can mean a later feed is never read at all.

## Output

One record per story, deduplicated across feeds.

| Field | Type | Notes |
|---|---|---|
| `feed` | string | Which feed reached the story first: `hottest`, `newest` or `t/<tags>`. Not a property of the story. |
| `shortId` | string | The site's short identifier. Stable, and the key to deduplicate and diff on. |
| `title` | string | Story title. |
| `url` | string \| null | The outbound link. `null` for discussion-only posts. |
| `isTextPost` | boolean | `true` exactly when `url` is `null`. |
| `score` | number \| null | Net score at run time. |
| `flags` | number \| null | Flag count at run time. |
| `commentCount` | number \| null | Comments at run time. |
| `createdAt` | string \| null | Submission time, ISO 8601 with the site's own offset, not UTC. |
| `submitterUser` | string \| null | Username of the submitter. |
| `userIsAuthor` | boolean \| null | Whether the submitter wrote the linked thing. Useful for separating self-promotion from curation. |
| `tags` | string[] | Every tag on the story. |
| `shortIdUrl` | string \| null | Short permalink to the story. |
| `commentsUrl` | string \| null | Full comments URL, with the slug. |
| `description` | string \| null | Submitter's note as HTML. `null` when there is none, which is most of the time. |
| `descriptionPlain` | string \| null | The same note as plain text. |
| `scrapedAt` | string | Run timestamp, ISO 8601. |

A real record from the front page:

```json
{
  "feed": "hottest",
  "shortId": "xx83dr",
  "title": "jujutsu 0.44.0",
  "url": "https://github.com/jj-vcs/jj/releases/tag/v0.44.0",
  "isTextPost": false,
  "score": 13,
  "flags": 0,
  "commentCount": 10,
  "createdAt": "2026-08-06T04:03:51.889-05:00",
  "submitterUser": "olex",
  "userIsAuthor": false,
  "tags": ["release", "vcs"],
  "shortIdUrl": "https://lobste.rs/s/xx83dr",
  "commentsUrl": "https://lobste.rs/s/xx83dr/jujutsu_0_44_0",
  "description": null,
  "descriptionPlain": null,
  "scrapedAt": "2026-08-06T11:43:10.687Z"
}
```

Where a submitter added a note, both description fields carry it:

```json
{
  "shortId": "kuopn0",
  "title": "Scripting, Programming Languages, Predictions",
  "description": "<p>by Roberto Ierusalimschy, creator of Lua</p>\n",
  "descriptionPlain": "by Roberto Ierusalimschy, creator of Lua",
  "tags": ["lua", "video"]
}
```

## RUN_SUMMARY

Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is
where a partial run admits that it was partial.

```json
{
  "feedsRequested": ["hottest", "newest", "t/rust,go"],
  "feedsFailed": 1,
  "failures": [
    { "feed": "t/nosuchtag", "page": 1, "error": "feed not found (404), check the tag name" }
  ],
  "storiesSaved": 87,
  "filters": { "tags": ["rust", "go"], "minScore": 5, "maxStories": 100, "maxPagesPerFeed": 4 },
  "finishedAt": "2026-08-06T11:43:12.041Z"
}
```

`feedsRequested` is the list of normalised feed names, not a count, so it is the place
to check that `rust,go` became the one combined feed you intended. `feedsFailed` counts
failure entries, and one feed can contribute several if it failed on more than one page,
so it can exceed the number of feeds. Each failure names the `page` it died on: a
failure on page 3 means pages 1 and 2 are in the dataset. `storiesSaved` equal to
`filters.maxStories` means the cap bound first and later feeds may not have been read.

## Behaviour to plan around

- **Stories are deduplicated across feeds by `shortId`.** A story on both `hottest` and
  `t/rust` is saved once, attributed to whichever feed reached it first. So `feed` tells
  you how the story was found, not where it lives, and per-feed counts will not add up
  to `storiesSaved` on overlapping feeds.
- **`score`, `flags` and `commentCount` are a snapshot at `scrapedAt`,** not settled
  values. A story an hour old is still climbing. Re-run and key on `shortId` to build a
  trajectory rather than treating one run as final.
- **`createdAt` carries the site's own offset, not UTC.** Convert before comparing
  against your own timestamps or bucketing by day.
- **A comma feed and separate feeds are not the same run.** `["rust,go"]` is one feed
  whose `feed` field is `t/rust,go` for every row. `["rust", "go"]` is two feeds, and a
  story tagged both is attributed to whichever was listed first. Pick based on whether
  you need to tell them apart.
- **`tags` filters on exact tag names, not substrings.** `web` does not match `webdev`.
  The story's own `tags` array is in every record, so filter afterwards if you want
  looser matching.
- **Paging stops early on a short page.** Fewer than 25 stories means the feed is
  exhausted, so `maxPagesPerFeed` is an upper bound and quiet tags will use less of it.
- **One bad feed never aborts the run.** A misspelled tag is a not-found, recorded in
  `failures`, and other feeds continue. The Actor only throws when every feed failed.
- **Transient errors are retried** three times with linear backoff. A not-found and an
  unexpected payload shape are final for that page and end that feed there.
- **`description` is usually `null`.** Roughly one story in twenty carries a submitter
  note; the empty string the source sends is normalised to `null`.

## Recipes

**Morning engineering digest.** Front page only, worth-reading threshold.

```json
{
  "feeds": ["hottest"],
  "minScore": 10,
  "maxStories": 50,
  "maxPagesPerFeed": 4
}
```

Sort on `score`, render `title`, `url` and `commentsUrl`. Keep the `shortId` set from
yesterday to suppress repeats.

**Niche tag watch.** A quiet tag needs depth, because the score filter discards most of
each page.

```json
{
  "feeds": ["t/compilers"],
  "minScore": 10,
  "maxStories": 200,
  "maxPagesPerFeed": 20
}
```

Stop on `createdAt` yourself once you reach material you already have.

**Release firehose.** Version announcements across the whole site.

```json
{
  "feeds": ["hottest", "newest"],
  "tags": ["release"],
  "minScore": 0,
  "maxStories": 100,
  "maxPagesPerFeed": 8
}
```

`url` usually points at a release page, so it maps cleanly onto a changelog feed.

**Cross-community overlap.** Broad, unfiltered, then joined elsewhere.

```json
{
  "feeds": ["hottest", "newest"],
  "maxStories": 300,
  "maxPagesPerFeed": 12
}
```

Normalise `url` (strip query strings and trailing slashes) before joining against
another aggregator; `shortId` is local to this site and will not match anything
external.