--- name: hacker-news-scraper description: Read Hacker News feeds as structured records via the Apify Actor arman-bd/hacker-news-scraper. Returns one row per item from the front page, newest, best, Ask HN, Show HN or the YC job board, with title, link, self-post text as clean plain text, score, author, posted timestamp, HN comment total and a permalink, plus an optional nested reply tree to a chosen depth. Use when a task needs launch and sentiment tracking, Ask HN mining for market research, a curated tech digest, or front-page position tracking on a schedule. Not for user profiles, keyword search across the archive, vote or karma history, or comments detached from their parent story. --- # Hacker News Scraper Apify Actor `arman-bd/hacker-news-scraper`. Pick one or more feeds, get one dataset record per item, optionally with its reply tree nested inside. It runs without credentials. ## When to use it - Watching Show HN and the front page for launches, and reading the discussion rather than just the headline. - Mining Ask HN threads, which are long-form problem statements from the audience most developer-tools companies sell to. - A scheduled tech digest: highest-scoring stories only, comments off. - Front-page position and score trajectory, diffed on `id` between runs. - Building a comment corpus for a topic, where the parent story is the unit you want to group by anyway. ## When not to use it - Keyword search across the HN archive. There is no query parameter here: you get the current contents of a feed and filter downstream. - User profiles, karma, submission history or a user's comments. - Historical front pages. Every feed is point-in-time, and there is no way to ask for yesterday. - Comments on their own. They come nested under the story that owns them, which is how HN models them. Flatten `comments` yourself if you need a flat table. - Anything below a story: poll options and individual comment permalinks are not addressable inputs. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/hacker-news-scraper').call({ feeds: ['topstories', 'showstories'], maxItems: 50, includeComments: true, commentDepth: 2, minScore: 0, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~hacker-news-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"feeds":["topstories"],"maxItems":25,"minScore":50,"includeComments":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/hacker-news-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `feeds` | string[] | no | `["topstories"]` | Any of `topstories`, `newstories`, `beststories`, `askstories`, `showstories`, `jobstories`. Short forms `top`, `new`, `best`, `ask`, `show`, `job` and `jobs` are normalised. Unknown names are dropped with a warning, not an error. | | `maxItems` | integer | no | `50` | Cap **per feed**, 0 to 500. `0` means every ID the feed lists. Top, new and best hold up to 500 IDs; Ask HN and Show HN about 200; jobs about 30. | | `includeComments` | boolean | no | `false` | Fetch replies and nest them under `comments`. | | `commentDepth` | integer | no | `2` | Reply levels to follow, clamped to 1 to 10. Only read when `includeComments` is on. | | `minScore` | integer | no | `0` | Drop items scoring below this. Applied after each item is fetched, not before. | **Comment fetching is the cost multiplier; `maxItems` is the only brake on it.** Each item costs its own request, and each comment costs one more, so 30 stories with comments at depth 2 is a normal run while 500 stories at depth 10 is not. `minScore` does not save you anything: the score only arrives with the item, so a selective floor makes the Actor walk further down the feed in windows until the cap is filled or the feed runs out. That combination is the right one on `newstories`, where most submissions sit at 1 or 2 points, but budget for it walking most of the 500 IDs. On `jobstories`, note that job posts carry a score of 1 by convention, so any floor above 1 silently empties the feed. ## Output One record per item saved, tagged with the feed it was first seen in. | Field | Type | Notes | |---|---|---| | `id` | number | HN item ID. Join and dedup key. | | `type` | string \| null | `story`, `job`, `poll` or `comment`. | | `title` | string \| null | Submission title. | | `url` | string \| null | The external link. `null` for Ask HN and other self-posts, whose content is in `text`. | | `hnUrl` | string | Permalink to the discussion. Always present, built from `id`. | | `text` | string \| null | Self-post body as plain text: HTML stripped, entities decoded, paragraphs kept as blank lines. `null` for link posts. | | `score` | number \| null | Points at the moment of scraping. `null` on item types that carry no score. | | `by` | string \| null | Submitter's username. | | `time` | number \| null | Unix seconds, exactly as HN reports it. | | `postedAt` | string \| null | The same moment as an ISO 8601 string. | | `descendants` | number \| null | HN's own comment total for the whole thread, at every depth. | | `commentCount` | number | How many comments **this run** actually retrieved. `0` when `includeComments` is off. | | `feed` | string | The feed this item came from. On a cross-feed duplicate, the first feed in your list wins. | | `comments` | object[] | Nested reply tree, `[]` when comments are off. Each node: `id`, `by`, `time`, `postedAt`, `text`, `depth`, `replies`. `depth` starts at 1. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, comment tree trimmed: ```json { "id": 49168622, "type": "story", "title": "How to Make a Nintendo 64 Game in 2026", "url": "https://phoboslab.org/log/2026/08/xibalba64-making-of", "hnUrl": "https://news.ycombinator.com/item?id=49168622", "text": null, "score": 137, "by": "atan2", "time": 1785849843, "postedAt": "2026-08-06T02:44:03.000Z", "descendants": 35, "commentCount": 12, "feed": "topstories", "comments": [ { "id": 49195265, "by": "FlavioMacedo", "time": 1786015785, "postedAt": "2026-08-06T11:29:45.000Z", "text": "Great work! I didn't even know it was still possible to publish new games for old consoles like the N64.", "depth": 1, "replies": [] } ], "scrapedAt": "2026-08-06T11:30:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "feedsRequested": ["topstories", "showstories"], "feedsIgnored": [], "feedsFailed": 0, "failures": [], "itemsSaved": 100, "commentsSaved": 1843, "duplicatesSkipped": 7, "filters": { "maxItems": 50, "minScore": 0, "includeComments": true, "commentDepth": 2 }, "finishedAt": "2026-08-06T11:34:12.417Z" } ``` `feedsIgnored` lists feed names you passed that were not recognised, so check it whenever you got fewer feeds back than you asked for. `failures` mixes two kinds of entry: a whole feed that could not be listed carries `feed` and `error`, while a single item that could not be fetched also carries `itemId`. Only the first kind counts towards `feedsFailed`, so `feedsFailed: 0` with a non-empty `failures` means individual items were lost and the feeds themselves were fine. `filters` shows the values after clamping, so compare `filters.commentDepth` against what you sent. ## Behaviour to plan around - **`descendants` and `commentCount` are different numbers and both are correct.** `descendants` is the whole thread as HN counts it; `commentCount` is what this run retrieved at your `commentDepth`. The gap is replies deeper than you asked for, plus anything deleted. Never use `commentCount` as a popularity signal. - **Cross-feed duplicates are saved once.** A story on both `topstories` and `beststories` appears one time, tagged with the first feed in your `feeds` order, and counted in `duplicatesSkipped`. If you need per-feed membership, run the feeds separately. - **`minScore` makes the run longer, not shorter.** The score is only known after the item is fetched, so the Actor walks the feed in windows until the cap is met. A high floor on `newstories` can mean fetching all 500 IDs to save 20 rows. - **Scores are point-in-time and move fast.** A front-page story's score changes by the minute. Re-run on a schedule and diff on `id` if you want the trajectory. - **Deleted and dead items are skipped**, at story level and inside comment trees, rather than being saved as empty records. That is a second reason `commentCount` can fall short of `descendants`. - **`text` is plain text, not HTML.** Entities are decoded before tags are stripped, so an encoded code sample survives as literal text. Paragraphs become blank lines. - **An unknown feed name is a warning, not an error.** It lands in `feedsIgnored` and the run continues on the rest. The Actor only throws when no valid feed remained, or when every listed feed failed to load. - **`commentDepth` is clamped to 1 to 10** and is ignored entirely when `includeComments` is off. A non-integer is floored. - **Job posts have no meaningful score.** They sit at 1. Keep `minScore` at 0 whenever `jobstories` is in the list. ## Recipes **Launch and sentiment watch.** Show HN plus the front page, with enough comment depth to read the reaction. ```json { "feeds": ["showstories", "topstories"], "maxItems": 100, "includeComments": true, "commentDepth": 2 } ``` Filter `title` and `text` for your product name, then read the `comments` tree for the matching rows. **Ask HN for market research.** Deeper trees, because the useful answers are usually replies to replies. ```json { "feeds": ["askstories"], "maxItems": 200, "includeComments": true, "commentDepth": 3, "minScore": 20 } ``` Flatten `comments` recursively on `replies` and keep `depth` as a column. **Cheap scheduled digest.** No comments, high floor, small cap. Finishes in seconds. ```json { "feeds": ["beststories", "topstories"], "maxItems": 50, "minScore": 150, "includeComments": false } ``` Deduplication across the two feeds is already done for you; sort by `score` and take the top slice. **Rule of thumb for cost.** Roughly one request per item, plus one per comment retrieved. Comments off, a 50-item feed is 50 requests. Comments on at depth 2 against a busy front page is closer to a thousand. Start with `includeComments: false` to see the item count, then turn it on for the subset you actually want to read.