--- name: bigcommerce-store-scraper description: Read public BigCommerce storefronts as a product catalogue via the Apify Actor arman-bd/bigcommerce-store-scraper. Returns one record per product with name, SKU, list price, sale price, currency, availability, breadcrumb categories, brand, images, plain-text description and the full configurable-option matrix including each option value's ID. Use when a task needs competitor price monitoring, assortment comparison across several merchants, or a supplier catalogue lifted into your own system. Not for orders, customers, inventory management, stores running on another platform, or per-variant price lists. --- # BigCommerce Store Scraper Apify Actor `arman-bd/bigcommerce-store-scraper`. Give it storefront domains, get one dataset record per product. It runs without credentials and reads only what a shopper's browser sees. ## When to use it - A competitor's catalogue has to become rows: SKU, list price, sale price, availability. - Several merchants in one run, compared on `categories` and `brand` to find range gaps. - A supplier's storefront section has to be lifted into your own system, options and option value IDs included. - Scheduled change detection, diffing `price`, `salePrice` and `availability` on `productId`. ## When not to use it - Orders, customers, carts, checkouts or inventory management. None of that is public and none of it is returned. - Stores on Shopify, WooCommerce, Squarespace or anything else. A non-BigCommerce host produces no products and lands in `RUN_SUMMARY.failures`. - Per-variant price lists. You get the base product price plus the option matrix; most merchants encode variant pricing in the option labels themselves. - A single product lookup. This Actor earns its keep sweeping a catalogue. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/bigcommerce-store-scraper').call({ stores: ['www.bulkapothecary.com'], maxProductsPerStore: 100, includeOptions: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); for (const p of items) console.log(p.sku ?? p.productId, p.salePrice ?? p.price, p.currency); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~bigcommerce-store-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"stores":["www.bulkapothecary.com"],"maxProductsPerStore":25,"includeOptions":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/bigcommerce-store-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `stores` | string[] | yes | | Storefront hosts or URLs. `www.bulkapothecary.com` and `https://www.bulkapothecary.com/some/page` are the same input: everything past the host is dropped. Duplicate hosts are removed. | | `categoryIds` | string[] | no | `[]` | Category page paths or full URLs, `fragrance-oils` or `/shop-all/kitchen/`. Empty means sweep the whole catalogue. Numeric BigCommerce category IDs do not work here, see below. | | `maxProductsPerStore` | integer | no | `50` | Per-store cap. `0` means no limit. Each product costs two requests, so a large cap on a big catalogue needs a longer run timeout. | | `includeOptions` | boolean | no | `true` | Include configurable options and modifiers. `false` drops the `options` key entirely and shrinks records a lot on stores with big option sets. | **`categoryIds` is not a filter, it is a different discovery route, and that is the one decision that matters.** Left empty, the Actor walks the store's public product sitemap, which every BigCommerce storefront serves regardless of theme: complete, slow, reliable. Filled in, it pages the category listings instead, which is faster and narrower but reads theme-standard product-card markup, so a merchant whose grid renders client-side returns nothing for that category. Sweep with it empty when you want the catalogue; use it only when you already know the merchant's category pages are server-rendered. The field keeps its name for compatibility but **numeric category IDs cannot be used**: public storefronts expose no id-addressable category route. Numeric entries are skipped with a warning and echoed back in `RUN_SUMMARY.filters.ignoredNumericCategoryIds`. ## Output One record per product that yielded a name. | Field | Type | Notes | |---|---|---| | `store` | string | Storefront host the record came from. Join back to your input on this. | | `productId` | number \| null | BigCommerce product ID, read from the add-to-cart form. The stable key for diffing across runs. | | `name` | string | Product name. | | `sku` | string \| null | Merchant SKU. `null` when the merchant publishes none. | | `price` | number \| null | **List price**: the MSRP or was-price when the store publishes one, otherwise the current price. | | `salePrice` | number \| null | **Discounted price actually charged**, or `null` when not on sale. What a shopper pays today is `salePrice ?? price`. | | `currency` | string \| null | ISO code for the storefront's active currency. | | `categories` | string[] | Breadcrumb trail with `Home` and the product itself stripped off. | | `brand` | string \| null | Only where the theme publishes one. | | `availability` | string \| null | `InStock`, `OutOfStock` or `PreOrder`. | | `inventoryLevel` | number \| null | Units in stock, only when the merchant enables public stock display. Usually `null`. | | `options` | object[] | Configurable options. Keys: `name`, `type` (the widget type, `set-radio`, `set-select`, …), `required`, `values` as `{ id, label }`. Key is absent when `includeOptions` is `false`. | | `images` | string[] | Full-resolution gallery URLs. | | `description` | string \| null | Description as plain text, HTML stripped and entities decoded. | | `url` | string | Canonical product URL. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "store": "www.bulkapothecary.com", "productId": 479, "name": "Clear Stick Lip Balm (Filled and Unlabeled)", "sku": "B850-016-000", "price": 21.84, "salePrice": null, "currency": "USD", "categories": ["Make It", "Lip Balm", "Filled and Unlabeled Product"], "brand": null, "availability": "InStock", "inventoryLevel": null, "options": [ { "name": "Flavor", "type": "set-radio", "required": true, "values": [ { "id": 75486, "label": "Original" }, { "id": 75484, "label": "Vanilla" } ] }, { "name": "Quantity", "type": "set-radio", "required": true, "values": [ { "id": 75478, "label": "Bag of 50 - ($0.44 / unit) - $21.84" }, { "id": 75479, "label": "Case of 1000 - ($0.35 / unit after discount) - $347.20" } ] } ], "images": ["https://cdn11.bigcommerce.com/s-99si0d/images/stencil/1280x1280/products/479/55514/Clear_Stick_Cherry_1__17136.1684366114.jpg?c=2"], "description": "Bulk Apothecary offers the best prices in the USA on unlabeled clear stick lip balm…", "url": "https://www.bulkapothecary.com/clear-stick-lip-balm-filled-and-unlabeled/", "scrapedAt": "2026-08-06T11:38:57.406Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a thin run explains itself. ```json { "storesRequested": 3, "storesFailed": 0, "perStore": [ { "store": "cornerstone-light-demo.mybigcommerce.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true }, { "store": "www.bulkapothecary.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true }, { "store": "www.berlinpackaging.com", "route": "sitemap", "discovered": 6, "saved": 6, "bigcommerceConfirmed": true } ], "failures": [], "productsSaved": 18, "filters": { "categoryIds": [], "ignoredNumericCategoryIds": [], "maxProductsPerStore": 6, "includeOptions": true }, "finishedAt": "2026-08-06T11:39:07.481Z" } ``` `perStore` separates the three ways a store comes back light. `discovered: 0` with `bigcommerceConfirmed` absent means the wrong platform or an unreachable host. `discovered` equal to your cap means you hit `maxProductsPerStore`, not the end of the catalogue. `discovered` well above `saved` means individual product pages failed, and each one is in `failures` with its URL. ## Behaviour to plan around - **`price` is not what the shopper pays.** It is the list price. The charged price is `salePrice ?? price`. A record with `salePrice: null` is simply not on sale. - **`categoryIds` changes the discovery route, not just the scope.** An empty array uses the sitemap and works on every theme. A filled array uses category listings and returns nothing on a store whose grid renders client-side. That case is logged, recorded in `failures`, and the run continues. - **Numeric category IDs are silently dropped**, not treated as an error. Check `filters.ignoredNumericCategoryIds` before concluding a filter had no effect. - **One failing store never aborts the run.** Discovery failures, non-BigCommerce hosts and per-product errors all land in `failures`. The Actor only throws when every store produced zero products. - **Sparse fields are honest, not broken.** `sku`, `brand` and `inventoryLevel` are `null` when the merchant does not publish them. `inventoryLevel` needs the merchant to have switched public stock display on, and most have not. - **`price: 0` is real.** Sample and quote-only products genuinely price at zero on some merchants. Do not treat zero as a missing value. - **`options` is absent, not `null`, when `includeOptions` is `false`.** Test for the key, not for a falsy value. - **Two requests per product, paced deliberately.** Budget the run timeout accordingly: `maxProductsPerStore: 0` on a catalogue of thousands is a long run, not a fast one. - **Transient errors are retried** three times with linear backoff. Not-found and forbidden responses are final for that URL and are not retried. ## Recipes **Daily price watch on one competitor.** Whole catalogue, options dropped so the diff stays clean and the records stay small. ```json { "stores": ["www.bulkapothecary.com"], "maxProductsPerStore": 0, "includeOptions": false } ``` Key on `store` plus `productId`, diff `price`, `salePrice` and `availability` against the previous run. **Assortment comparison across merchants.** Several stores in one run, capped so no single large catalogue dominates the dataset. ```json { "stores": [ "www.bulkapothecary.com", "https://www.berlinpackaging.com", "cornerstone-light-demo.mybigcommerce.com" ], "maxProductsPerStore": 500 } ``` Group by `brand` and by the last element of `categories` to see who carries what. **Supplier catalogue extraction.** Narrow to the categories you resell, keep options so each variant's `values[].id` comes across for mapping. ```json { "stores": ["www.bulkapothecary.com"], "categoryIds": ["fragrance-oils", "essential-oils"], "maxProductsPerStore": 0, "includeOptions": true } ``` If a category returns nothing, check `failures` for it and re-run that store with `categoryIds` empty; the sitemap route does not depend on the theme.