--- name: rss-feed-scraper description: Read any RSS, Atom or JSON feed into one normalised record shape via the Apify Actor arman-bd/rss-feed-scraper. Returns per item the feed title and format, the feed's own item ID, headline, absolute permalink, author, ISO 8601 publish and update timestamps, plain-text summary and full body, optional original markup, categories, media enclosures and a lead image, with the feed auto-discovered from a plain site homepage. Use for newsroom and competitor monitoring, news aggregation across many publishers, keyword-filtered curation pipelines and article archiving. Not for sites without a feed, paginated archives older than the feed window, authenticated feeds or full-site crawling. --- # RSS & Atom Feed Scraper: Universal Content Monitor Apify Actor `arman-bd/rss-feed-scraper`. Give it feed URLs, or plain site homepages, and get one dataset record per item in a single shape regardless of which syndication format the publisher used. It runs without credentials. ## When to use it - Watching dozens of blogs or newsrooms on a schedule and diffing on `itemId`. - Aggregating many publishers where downstream code needs one record shape, not five. - Keyword-filtered ingest, where only items mentioning your terms should reach the next step. - Archiving full article bodies before publishers rewrite or unpublish them. - Reading a mixed list of RSS, Atom and JSON feeds in one run without special-casing any. ## When not to use it - Sites that publish no feed. Discovery is thorough but it cannot invent one, and the source is recorded as a failure. - History older than the feed window. Most publishers expose their 10 to 50 most recent items and nothing else; this is not an archive crawler. - Authenticated, private or paywalled feeds. Those come back as an access failure. - Crawling a site's pages. Only the feed document and its items are read. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/rss-feed-scraper').call({ feedUrls: ['https://arstechnica.com', 'https://blog.rust-lang.org/feed.xml'], maxItemsPerFeed: 25, sinceDate: '2026-08-01', keywordFilter: [], includeContent: true, stripHtmlTags: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~rss-feed-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"feedUrls":["https://news.ycombinator.com/rss","https://github.blog"],"maxItemsPerFeed":25,"includeContent":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/rss-feed-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `feedUrls` | string[] | yes | | Feed URLs or site homepages. A bare host such as `github.blog` gets `https://` prepended. Entries that are not valid URLs are dropped, and duplicates are removed. If the URL is not itself a feed, the page's advertised alternate links are followed, and failing that the conventional paths `/feed`, `/rss`, `/rss.xml`, `/feed.xml`, `/atom.xml`, `/index.xml` are tried in order. | | `maxItemsPerFeed` | integer | no | `0` | Cap on saved items per feed, applied after filtering, 0 to 10000. `0` means no cap, which is usually right because a feed only carries a small window anyway. | | `sinceDate` | string | no | `""` | Keep items published after this moment. A plain date (`2026-01-01`) or a full ISO 8601 timestamp both work. An unparseable value throws before any request. | | `keywordFilter` | string[] | no | `[]` | Keep items whose title, summary or categories contain one of these terms, case-insensitive and OR-combined. Matching is on the summary, not the full body. | | `includeContent` | boolean | no | `true` | Populate `contentText` with the full body. Off leaves both body fields `null` and makes the dataset far smaller. | | `stripHtmlTags` | boolean | no | `true` | On, bodies are plain text and `contentHtml` is `null`. Off additionally keeps the publisher's original markup in `contentHtml`. Has no effect when `includeContent` is off. | **Filters combine with AND, and both run before the cap.** An item must pass `sinceDate` and match at least one keyword, and only what survives is counted against `maxItemsPerFeed`, so a cap of 20 with a keyword filter gives you 20 matching items rather than 20 items of which some match. The cheap-versus-complete decision is `includeContent`: off is the right setting for change detection sweeps across many feeds, on with `stripHtmlTags` off is the right setting when you are archiving and need the images and links inside the article preserved. ## Output One record per item saved, carrying its feed's identity so a multi-feed run stays joinable. | Field | Type | Notes | |---|---|---| | `requestedUrl` | string | Exactly the normalised URL you passed. Join back to your input list on this. | | `feedUrl` | string | The feed actually read, after discovery and redirects. Differs from `requestedUrl` whenever you passed a homepage. | | `feedTitle` | string \| null | Channel title. | | `feedType` | string | `rss`, `rss-1.0`, `atom` or `json`. Useful for explaining field-level gaps. | | `itemId` | string \| null | The feed's own identity for the item: the RSS guid, Atom id or JSON Feed id, falling back to the link. The de-duplication and change-detection key. | | `title` | string \| null | Headline. | | `link` | string \| null | Permalink, resolved to an absolute URL against the feed. | | `author` | string \| null | Whichever author element the feed uses. Frequently `null`; many feeds simply omit it and nothing is guessed. | | `publishedAt` | string \| null | ISO 8601 UTC, whatever format the feed wrote. `null` when the feed gives no date or the date is unreadable. | | `updatedAt` | string \| null | ISO 8601 UTC. Usually only Atom feeds set it. | | `summary` | string \| null | Short description as plain text, always stripped and decoded. | | `contentText` | string \| null | Full body as plain text. `null` when `includeContent` is off, and also `null` when the feed carries no body. | | `contentHtml` | string \| null | The publisher's original markup. Non-null only when `includeContent` is on and `stripHtmlTags` is off. | | `categories` | string[] | Tags the item is filed under, de-duplicated. `[]` when none. | | `enclosures` | object[] | Attached media as `[{ url, type, length }]`: podcast audio, video, PDFs. `length` is bytes or `null`. `[]` when none. | | `imageUrl` | string \| null | Lead image, taken in order from a media element, a thumbnail, a podcast image, an image enclosure, or the first image in the body. Absolute. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record from a homepage that was resolved to its feed: ```json { "requestedUrl": "https://arstechnica.com/", "feedUrl": "https://arstechnica.com/feed/", "feedTitle": "Ars Technica", "feedType": "rss", "itemId": "https://arstechnica.com/gadgets/2026/08/ai-isnt-enough-to-protect-social-media-communities-from-ai/", "title": "AI isn't enough to protect social media communities from AI", "link": "https://arstechnica.com/gadgets/2026/08/ai-isnt-enough-to-protect-social-media-communities-from-ai/", "author": "Scharon Harding", "publishedAt": "2026-08-06T11:00:44.000Z", "updatedAt": null, "summary": "Why humans need to moderate humans.", "contentText": "Sometimes you have to fight fire with fire. But when it comes to AI slop and hateful content threatening the safety and value of social media platforms …", "contentHtml": null, "categories": ["AI", "Tech", "discord", "Facebook", "generative ai", "reddit", "social media"], "enclosures": [], "imageUrl": "https://cdn.arstechnica.net/wp-content/uploads/2026/07/GettyImages-2259634870-1152x648.jpg", "scrapedAt": "2026-08-06T11:43:41.586Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "sourcesRequested": 3, "sourcesFailed": 1, "failures": [ { "source": "https://example.com/", "error": "no readable feed found (not a feed (root element ))" } ], "itemsSaved": 47, "perFeed": [ { "source": "https://arstechnica.com/", "feedUrl": "https://arstechnica.com/feed/", "feedType": "rss", "listed": 20, "saved": 20 } ], "filters": { "maxItemsPerFeed": null, "sinceDate": "2026-08-01T00:00:00.000Z", "keywordFilter": ["election", "budget"], "includeContent": false, "stripHtmlTags": true }, "finishedAt": "2026-08-06T11:43:45.902Z" } ``` `perFeed` is the diagnostic that matters. `listed` is how many items the feed carried and `saved` is how many survived your filters and the cap, so a large gap means your filters are doing the work, not that anything was lost. A source in `failures` produced no `perFeed` entry at all; the error text distinguishes a dead host, a not-found, an access refusal and a page with no discoverable feed. `filters.maxItemsPerFeed` is `null` when uncapped, and `filters.sinceDate` is the parsed ISO form of whatever you sent, worth checking if a date filter behaved unexpectedly. A run where every source fails throws. ## Behaviour to plan around - **A feed is a window, not an archive.** Publishers typically expose 10 to 50 recent items. Asking for more does not produce more; only running on a schedule accumulates history. - **`contentText` equal to `summary` is normal.** Many feeds, especially link blogs and aggregators, publish only a description and no body, so the body reader falls back to the same element. Nothing is being dropped. - **`contentHtml` needs two flags.** It is `null` unless `includeContent` is on and `stripHtmlTags` is off. Turning off `stripHtmlTags` alone while `includeContent` is off gives you nothing. - **`sinceDate` never drops undated items.** An item whose feed gives no publish date passes the date filter regardless. Filter on `publishedAt !== null` yourself if you need strictness. - **Discovery follows the first advertised feed.** A site with a main feed and several category feeds gives you the main one. Pass the specific URL if you want a category. Conventional paths are only tried when the page advertises nothing at all. - **`feedUrl` is the row's real provenance.** After discovery and redirects it can differ from `requestedUrl` in ways that matter, such as a homepage resolving to a comments feed on a badly configured site. Check it before trusting a source label. - **De-duplication is per feed, not global.** The same story syndicated by two publishers is saved twice, correctly. De-duplicate on `link` across feeds yourself if you need to. - **Dates are normalised or `null`, never guessed.** Both RFC 822 and ISO 8601 come out as ISO 8601 UTC, and an unreadable date becomes `null` rather than a plausible wrong value. - **Sources are read one per second** with a generous per-request timeout, because feeds live on other people's servers. A long list is a long run; size the run timeout for it. - **Transient failures are retried** three times with backoff, honouring a server's own retry hint. A not-found and an access refusal are final for that source. ## Recipes **Multi-format smoke test.** Three feeds, three formats, one record shape. ```json { "feedUrls": [ "https://news.ycombinator.com/rss", "https://blog.rust-lang.org/feed.xml", "https://daringfireball.net/feeds/json" ], "maxItemsPerFeed": 0, "includeContent": true, "stripHtmlTags": true } ``` Group by `feedType` to see how field coverage differs between formats. **Scheduled change detection across many homepages.** Bodies off, so it stays cheap. ```json { "feedUrls": ["https://arstechnica.com", "github.blog", "https://blog.rust-lang.org/"], "maxItemsPerFeed": 20, "includeContent": false, "stripHtmlTags": true } ``` Keep the set of `itemId` values from the previous run and emit only what is new. **Keyword-filtered curation.** Both filters apply before the cap, so the cap counts matches. ```json { "feedUrls": ["https://feeds.bbci.co.uk/news/rss.xml"], "sinceDate": "2026-08-01", "keywordFilter": ["election", "budget"], "maxItemsPerFeed": 50, "includeContent": false } ``` **Archival capture with markup intact.** Keeps images and links inside the body. ```json { "feedUrls": ["https://blog.rust-lang.org/feed.xml"], "maxItemsPerFeed": 0, "includeContent": true, "stripHtmlTags": false } ``` Store `contentHtml` for fidelity and `contentText` for indexing; both are present in this mode.