--- name: product-feed-scraper description: Parse public RSS 2.0 product feeds, including Google Merchant feeds using the g namespace, as structured records via the Apify Actor arman-bd/product-feed-scraper. Returns one record per product with id, variant group id, title, plain-text description, product and image links, price and sale price split into a number and a currency code, availability as an enum and a boolean, brand, GTIN, MPN, condition, merchant and Google categories, and the shipping group. Use for competitive price monitoring, supplier or dropship catalogue ingest, GTIN-keyed comparison shopping, stock and assortment tracking or sector price research. Not for Atom feeds, CSV or TSV feeds, HTML storefront pages, feeds that are not publicly served, or currency conversion. --- # Product Feed Scraper Apify Actor `arman-bd/product-feed-scraper`. Give it the URLs of public product feeds, get one dataset record per product. These feeds exist to be read by machines: a merchant generates one deliberately and hands the URL to an ad platform, and this Actor reads the same URL. It runs without credentials; you supply nothing but the list. ## When to use it - Daily competitive price monitoring, diffing on `productId` plus `price`. - Ingesting a supplier's, distributor's or dropship partner's whole catalogue into your own system. - Comparison shopping across several merchants, normalised into one schema and keyed on `gtin`. - Stock and assortment tracking: watching `availability` flip across a competitor's range. - Price distribution research by brand, category or condition across a sector. ## When not to use it - Atom feeds. They are detected and rejected with a clear message rather than parsed into empty rows. - CSV or TSV feeds. This is an XML reader. - Scraping a storefront's HTML. If a merchant publishes no feed, there is nothing here to read. - Feeds that are not publicly served. Those come back as an access-denied failure. - Currency conversion, price history or ranking. You get the number the feed states, at the moment you read it. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/product-feed-scraper').call({ feedUrls: ['https://example-shop.com/feeds/google-shopping.xml'], maxProductsPerFeed: 500, inStockOnly: true, brandFilter: [], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~product-feed-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"feedUrls":["https://example-shop.com/feeds/google-shopping.xml"],"inStockOnly":true,"maxProductsPerFeed":1000}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/product-feed-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `feedUrls` | string[] | yes | | Public product feed URLs. A missing scheme is filled in, redirects are followed, and duplicates are removed. A compressed feed is decompressed automatically, whether the compression is in the transfer encoding or in the file body itself. | | `maxProductsPerFeed` | integer | no | `0` (no limit) | Cap on products **saved** per feed, counted after the filters below. Catalogues run to tens of thousands of items, so set this while exploring a feed you have not seen. | | `inStockOnly` | boolean | no | `false` | Keep only products whose availability normalises to `in_stock`. Feeds write it with a space or an underscore and both are recognised. A product that states no availability is dropped by this filter, because nothing says it is buyable. | | `brandFilter` | string[] | no | `[]` | Keep only products whose brand exactly matches one of these, case-insensitively. A product with no brand is dropped when this is set. | | `minPrice` | integer | no | unset | Lower price bound, in the feed's own currency. No conversion is done. | | `maxPrice` | integer | no | unset | Upper price bound, in the feed's own currency. A `minPrice` above `maxPrice` aborts the run before any request. | **The filters run before anything is written, so they cut the dataset, not just your view of it.** They combine with AND, and each one has a defined behaviour on missing data that decides how aggressive it is: setting either price bound drops every product whose price could not be parsed, `inStockOnly` drops every product with no availability field, and `brandFilter` drops every product with no brand, which is common among small merchants. If a run comes back far emptier than the catalogue, compare `RUN_SUMMARY.perFeed[].listed` against `saved` before assuming the feed is small. ## Output One record per product that survived the filters. | Field | Type | Notes | |---|---|---| | `feedUrl` | string | The feed URL **after redirects**, so it can differ from what you passed in. | | `feedTitle` | string \| null | Channel title, usually the store name. | | `productId` | string \| null | The merchant's product id. Falls back to the item guid, then to the link, so it is not always a stable sku. | | `itemGroupId` | string \| null | Ties variants of one product together. Equal to `productId` on feeds that do not use variants. | | `title` | string \| null | Product title. | | `description` | string \| null | Description with HTML stripped and line breaks preserved. `null` when the feed carries none. | | `link` | string \| null | Product page URL. | | `imageLink` | string \| null | Main image. Falls back through a media element, an enclosure, and finally the first inline image in the description. | | `additionalImageLinks` | string[] | Extra images, deduplicated. Often empty. | | `price` | number \| null | List price as a number. `null` when unparseable. | | `salePrice` | number \| null | Sale price as a number, when the feed states one. | | `currency` | string \| null | ISO code for both prices. Taken from the list price, falling back to the sale price. | | `availability` | string \| null | Normalised to underscores and lower case: `in_stock`, `out_of_stock`, `preorder`, `backorder`. | | `inStock` | boolean \| null | `true` only when `availability` is `in_stock`. `null` when the feed states no availability, which is not the same as `false`. | | `brand` | string \| null | Brand as the merchant wrote it, often the store's own name. | | `gtin` | string \| null | GTIN, accepting EAN, UPC and ISBN as aliases. `null` for the many merchants who leave it blank. | | `mpn` | string \| null | Manufacturer part number. | | `condition` | string \| null | Normalised: `new`, `used`, `refurbished`. | | `productType` | string \| null | The merchant's own taxonomy, usually a `>` separated path. | | `googleProductCategory` | string \| null | The ad platform's category, as an id or a path depending on the generator. | | `shippingCost` | number \| null | From the nested shipping group. | | `shippingCurrency` | string \| null | Currency of `shippingCost`, parsed separately from the product price. | | `shippingCountry` | string \| null | Country the shipping group applies to. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record: ```json { "feedUrl": "https://houseofjerky.com/wp-content/uploads/woo-feed/google/xml/googlehojfeed.xml", "feedTitle": "House of Jerky", "productId": "97864", "itemGroupId": "97864", "title": "Teriyaki Lovers", "description": null, "link": "https://houseofjerky.com/shop-for-jerky/teriyaki-lovers/", "imageLink": "https://houseofjerky.com/wp-content/uploads/2025/10/Teriyaki-Lovers-scaled.jpg", "additionalImageLinks": [], "price": 69.99, "salePrice": 69.99, "currency": "USD", "availability": "in_stock", "inStock": true, "brand": "House of Jerky", "gtin": null, "mpn": "teriyakilovers", "condition": "new", "productType": "Beef > Exotic > Game > Turkey > Subscriptions", "googleProductCategory": "423", "shippingCost": null, "shippingCurrency": null, "shippingCountry": null, "scrapedAt": "2026-08-06T11:56:59.217Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** `perFeed` is the only place that distinguishes a small catalogue from an aggressive filter. ```json { "feedsRequested": 3, "feedsFailed": 1, "failures": [ { "feedUrl": "https://example-shop.com/collections/all.atom", "error": "this is an Atom feed, only RSS 2.0 product feeds are supported" } ], "productsSaved": 812, "perFeed": [ { "feedUrl": "https://example-shop.com/feeds/google-shopping.xml", "feedTitle": "Example Shop", "listed": 4210, "saved": 500, "filtered": 118 }, { "feedUrl": "https://another-shop.com/googlebase.xml", "feedTitle": "Another Shop", "listed": 402, "saved": 312, "filtered": 90 } ], "filters": { "maxProductsPerFeed": 500, "inStockOnly": true, "brandFilter": [], "minPrice": null, "maxPrice": null }, "finishedAt": "2026-08-06T11:57:03.884Z" } ``` Read each `perFeed` row as three numbers that need not add up. `listed` is every item in the feed. `saved` plus `filtered` equals `listed` only when the run walked the whole feed; when `maxProductsPerFeed` bound first, the loop stopped early and the remainder was never examined. So `saved` equal to `filters.maxProductsPerFeed` means truncation, and `saved` well below `listed` with a large `filtered` means your filters did it. `failures` names feeds that were never read at all. ## Behaviour to plan around - **`feedUrl` on a record is the post-redirect URL.** `failures[].feedUrl` is the URL you asked for. Joining a run back to your input list on `feedUrl` will miss any feed that redirected; use `perFeed` order or your own list instead. - **`inStock: null` is not `false`.** It means the feed stated no availability. Filter on `inStock === true` for buyable, and treat `null` as unknown rather than out of stock. - **A bare dollar sign is read as USD.** `C$`, `A$`, `NZ$` and `R$` are recognised first, but every other dollar currency written with a plain `$` will be labelled USD. An explicit three-letter code in the price string always wins. Check `currency` against the merchant's country before comparing prices across feeds. - **Prices parse from several layouts.** A trailing or leading ISO code, a symbol prefix and continental separators are all handled, so `1.299,00 EUR` and `£1,050` both come out right. A price that still cannot be read is `null`, never zero. - **`productId` is only as good as the feed.** It falls back to the item guid and then to the product link. Two merchants can use the same value, so key cross-merchant joins on `gtin` and use `feedUrl` plus `productId` within one merchant. - **A feed with no namespaced merchant fields still parses.** It falls back to plain RSS title, link and description, and the run log warns that this happened. Rows will have `price`, `brand`, `gtin` and the rest as `null`, which usually means a blog feed was pasted in by mistake. - **Non-UTF-8 catalogues are decoded** from the declaration in the document, so accented titles from European carts survive intact. - **One bad feed never aborts the run.** A not-found, an access-denied, an Atom feed or anything that is not RSS is recorded in `failures` and the next feed is read. The Actor only throws when every feed failed. - **Transient errors are retried** three times with linear backoff, honouring a retry-after hint where one is given. Not-found, access-denied and wrong-format responses are final for that feed. - **The whole feed is held in memory while parsing.** A very large catalogue needs the run's memory raised; the request timeout is already generous because a hundred-thousand-product feed is tens of megabytes. - **There is no deduplication between feeds.** The same product sold by two merchants is two rows, which is usually what you want for comparison. ## Recipes **Daily price monitor.** The whole catalogue, no filters, one snapshot per day. ```json { "feedUrls": ["https://example-shop.com/feeds/google-shopping.xml"], "maxProductsPerFeed": 0, "inStockOnly": false } ``` Diff against yesterday on `productId` plus `price` and `salePrice`. Flag any `availability` change separately; those are assortment moves, not price moves. **Explore a feed you have not seen.** Cap it, look at the shape, then widen. ```json { "feedUrls": ["https://example-shop.com/googlebase.xml"], "maxProductsPerFeed": 200 } ``` Check how many rows have a non-null `gtin` and `brand` before designing any join on them, and read `perFeed[].listed` for the true catalogue size. **Buyable products from one brand in a price band.** Filters run before the write, so this is cheap. ```json { "feedUrls": ["https://example-shop.com/feeds/google-shopping.xml"], "inStockOnly": true, "brandFilter": ["Sony"], "minPrice": 100, "maxPrice": 900 } ``` Expect fewer rows than you would from filtering afterwards: no-brand and no-price products are dropped by these bounds. **Multi-merchant comparison table.** Several feeds, one run. ```json { "feedUrls": [ "https://example-shop.com/feeds/google-shopping.xml", "https://another-shop.com/googlebase.xml", "https://third-shop.com/google_shopping.xml" ], "inStockOnly": true, "maxProductsPerFeed": 5000 } ``` Group on `gtin`, discard the `null` group, and compare `price` within each group only after confirming `currency` matches.