--- name: npm-package-scraper description: Read npm registry documents as flat structured records via the Apify Actor arman-bd/npm-package-scraper. Returns the latest version, dist-tag map, licence, description, keywords, author and maintainers, cleaned repository, homepage, bugs and funding links, the four dependency maps for the latest version, engines, unpacked size, tarball URL, version and deprecation counts, the full publish timeline and an optional per-version history. Use when a task needs SBOM or licence compliance, dependency drift checks, release cadence, upgrade-risk review or an internal package catalogue. Not for download counts or popularity trends, and not for private packages. --- # npm Package Scraper: Metadata, Versions & Dependencies Apify Actor `arman-bd/npm-package-scraper`. Give it a list of package names, get one dataset record per package, flattened out of the registry document that `npm install` itself reads. Scoped names work exactly like plain ones. You supply no credentials. ## When to use it - A dependency list has to become rows: licence, repository, maintainers, latest version. - SBOM or licence compliance across everything a project pulls in. - Dependency drift: what you pin against what the registry ships today. - Release cadence and upgrade risk: publish timeline, deprecation state, `engines`. - An internal catalogue: descriptions, keywords and maintainers for a searchable index. ## When not to use it - Download counts, popularity or trends. None are in the registry document; use `arman-bd/npm-download-stats-scraper` for those. - Package search or discovery. There is no query input, only an explicit name list. - Private or scoped-internal packages. There is no token input. - Recursive dependency resolution. One run is one level; see the recipe below. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/npm-package-scraper').call({ packages: ['express', 'lodash', '@babel/core'], includeDependencies: true, includeAllVersions: false, includeDeprecated: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~npm-package-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"packages":["express","@babel/core"],"includeDependencies":true,"includeAllVersions":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/npm-package-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `packages` | string[] | yes | | Package names, scoped (`@babel/core`) or plain (`express`), or a full npmjs.com package link, which is reduced to the name. Duplicates are removed before any request. Malformed entries are dropped silently. | | `includeAllVersions` | boolean | no | `false` | Fill the `versions` array with every published release, its publish date, licence, dependency count and deprecation note. Off by default: a mature package can have hundreds. | | `includeDependencies` | boolean | no | `true` | Include the four dependency maps for the latest version. Off makes them `null`, not empty objects. | | `includeDeprecated` | boolean | no | `false` | Keep deprecated releases inside `versions`. No effect at all unless `includeAllVersions` is on. | **None of the flags change the cost of the run, only the size of the record.** One package is exactly one request whatever you set, because the registry serves the whole document at once. So the decision is about what you then have to store and parse: `includeAllVersions` can multiply a record's size by two orders of magnitude, and it is the only flag worth thinking about on a long list. Note the counts you probably want are free: `versionCount` and `deprecatedVersionCount` are the true totals regardless of the flags, so you can size a package's history without fetching it. ## Output One record per package that resolved. | Field | Type | Notes | |---|---|---| | `name` | string | Canonical registry name. Join on this. | | `latestVersion` | string \| null | The `latest` dist-tag, falling back to the last key in the version map. | | `description` | string \| null | Registry description, falling back to the latest version's. | | `license` | string \| null | Licence identifier, usually SPDX. `null` when the package declares none. | | `homepage` | string \| null | Project site. | | `repository` | string \| null | Source repository, normalised: the `git+` prefix, `git://` and `ssh://git@` schemes and the trailing `.git` are stripped, so it is directly linkable and joins against GitHub data. | | `bugs` | string \| null | Issue tracker URL. | | `author` | string \| null | Original author's name only, never the email or URL. | | `maintainers` | string[] | Current maintainer handles. Empty array when none are published. | | `keywords` | string[] | Keyword list. | | `distTags` | object | Full `{ tag: version }` map: `latest`, `next`, `latest-4`, whatever the project publishes. | | `dependencies` | object \| null | `{ name: range }` for the latest version. `null` when `includeDependencies` is off. | | `devDependencies` | object \| null | Same shape and same rule. | | `peerDependencies` | object \| null | Same shape and same rule. | | `optionalDependencies` | object \| null | Same shape and same rule. | | `dependencyCount` | number | Runtime dependency count for the latest version. Always present, even with `includeDependencies` off. | | `engines` | object \| null | e.g. `{ "node": ">= 18" }`. `null` when unspecified. | | `funding` | string \| null | Funding URL. | | `unpackedSizeBytes` | number \| null | Install size of the latest version. `null` on older releases that predate the field. | | `tarball` | string \| null | Tarball URL for the latest version. | | `versionCount` | number | Every version ever published. True total, flags do not affect it. | | `deprecatedVersionCount` | number | How many of those are deprecated. Also a true total. | | `versions` | object[] | Per-version history when `includeAllVersions` is on, each `{ version, published, license, dependencyCount, deprecated }`. **`[]` when the flag is off**, never `null`. | | `firstPublished` | string \| null | When the package was created, ISO 8601. | | `lastPublished` | string \| null | Last registry change of any kind, ISO 8601. Moves on a deprecation or a retag, not only on a publish. | | `latestPublished` | string \| null | When the current `latest` version actually shipped. This is the release-cadence field. | | `isDeprecated` | boolean | Whether the *latest* version is deprecated. | | `deprecationMessage` | string \| null | The maintainer's deprecation text, verbatim, when there is one. | | `url` | string | npmjs.com page for the package. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record for `express`, dependency and version lists truncated: ```json { "name": "express", "latestVersion": "5.2.1", "description": "Fast, unopinionated, minimalist web framework", "license": "MIT", "homepage": "https://expressjs.com/", "repository": "https://github.com/expressjs/express", "bugs": "https://github.com/expressjs/express/issues", "author": "TJ Holowaychuk", "maintainers": ["wesleytodd", "jonchurch", "ctcpip", "ulisesgascon", "sheplu"], "keywords": ["express", "framework", "web", "http", "rest", "router", "api"], "distTags": { "latest": "5.2.1", "latest-4": "4.22.2" }, "dependencies": { "body-parser": "^2.2.0", "content-type": "^1.0.5", "serve-static": "^2.2.0" }, "devDependencies": { "cookie-parser": "^1.4.7", "method-override": "^3.0.0" }, "peerDependencies": {}, "optionalDependencies": {}, "dependencyCount": 28, "engines": { "node": ">= 18" }, "funding": "https://opencollective.com/express", "unpackedSizeBytes": 75429, "tarball": "https://registry.npmjs.org/express/-/express-5.2.1.tgz", "versionCount": 288, "deprecatedVersionCount": 173, "versions": [ { "version": "5.2.1", "published": "2025-12-01T20:49:43.268Z", "license": "MIT", "dependencyCount": 28, "deprecated": null } ], "firstPublished": "2010-12-29T19:38:25.450Z", "lastPublished": "2026-07-14T18:49:18.499Z", "latestPublished": "2025-12-01T20:49:43.268Z", "isDeprecated": false, "deprecationMessage": null, "url": "https://www.npmjs.com/package/express", "scrapedAt": "2026-08-06T11:42:46.722Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "packagesRequested": 3, "packagesSaved": 2, "packagesFailed": 1, "failures": [ { "package": "expresss", "error": "package \"expresss\" not found (404)" } ], "filters": { "includeAllVersions": false, "includeDependencies": true, "includeDeprecated": false }, "finishedAt": "2026-08-06T11:42:47.005Z" } ``` `packagesRequested` is the count *after* normalisation and de-duplication, so it can be below the length of your input list without anything having gone wrong. Every shortfall between it and `packagesSaved` is itemised in `failures`, one `{ package, error }` per name. `filters` echoes the flags, which is how you tell an empty `versions` array caused by `includeAllVersions: false` from one caused by deprecation filtering. ## Behaviour to plan around - **`lastPublished` and `latestPublished` are different questions.** `lastPublished` is the registry's last-modified time and moves when a maintainer deprecates or retags something, with no new code. `latestPublished` is when the current `latest` version shipped. Use the second for release cadence, or a dormant project will look active. - **`versions: []` is ambiguous on its own.** Either `includeAllVersions` was off, or it was on and every release is deprecated while `includeDeprecated` was off. That second case is real for retired packages. `versionCount` always shows the true total, so compare against it rather than concluding the package has no releases. - **Dependency maps go `null`, not `{}`, when the flag is off.** An empty object is a genuine "this package has no dependencies of that kind". Do not conflate them. - **`dependencyCount` counts runtime dependencies only**, not dev, peer or optional, and it is computed even when the maps are suppressed. - **The maps are one level deep.** They are the latest version's declared ranges, not a resolved tree and not transitive. Building a tree means feeding the keys back in. - **The record describes the latest version, not the one you use.** `license`, `engines`, `unpackedSizeBytes`, `isDeprecated` and the dependency maps all come from `latestVersion`. If you pin an older release, the per-version `versions` array is the only place its licence and dependency count appear. - **A missing package never aborts the run.** An unpublished or misspelled name is final for that package, lands in `failures` and the run continues. The Actor only throws when every package failed. - **Transient failures are retried** three times with linear backoff. A not-found is not retried, because it will not become a success. - **`author` is a name, not a contact.** Emails and URLs from the registry document are deliberately not carried through. ## Recipes **Licence and compliance sweep.** Compact records, no history, dependency maps kept because they are what you audit next. ```json { "packages": ["express", "lodash", "@babel/core", "react", "typescript"], "includeDependencies": true, "includeAllVersions": false } ``` Flag every row where `license` is null or `isDeprecated` is true, then take the union of `dependencies` keys as the input to the next level. **Release cadence and maintenance health.** Full history, deprecated releases kept so the timeline has no holes. ```json { "packages": ["@babel/core", "typescript", "vite"], "includeAllVersions": true, "includeDeprecated": true } ``` Sort `versions` by `published` and difference consecutive dates for a release interval; use `latestPublished`, not `lastPublished`, for "how long since the last release". **Compact catalogue over a long list.** Nothing but the headline fields. ```json { "packages": ["react", "react-dom", "https://www.npmjs.com/package/next"], "includeDependencies": false, "includeAllVersions": false } ``` Index on `name`, `description`, `keywords` and `maintainers`. `versionCount` and `deprecatedVersionCount` still arrive, so you can rank by history depth without paying for it. **Two-level dependency tree.** Run the roots, collect the keys, run again. ```json { "packages": ["express"], "includeDependencies": true } ``` Take `Object.keys(dependencies)` from the result and pass that array as `packages` in a second run. Two passes cover almost every real tree; deeper needs a loop of your own.