--- name: crates-io-scraper description: Read Rust crate registry metadata from crates.io as structured records via the Apify Actor arman-bd/crates-io-scraper. Returns description, latest version, all-time and 90-day download counts, categories, keywords, licence, repository and documentation links and version count per crate, plus optional full publish history, the direct dependency graph of the default version and the list of publishers. Use when a task needs Rust ecosystem adoption tracking, crate selection on real usage, licence audits or supply-chain review across a Cargo.lock. Not for crate source, READMEs, transitive dependency resolution or anything outside the Rust registry. --- # crates.io Scraper Apify Actor `arman-bd/crates-io-scraper`. Give it crate names, get one dataset record per crate. It runs without credentials. Three independent toggles decide how deep each record goes. ## When to use it - A fixed crate list has to become rows: downloads, latest version, licence, last publish. - Choosing between competing crates on measured adoption rather than stars. - Licence audit or supply-chain review over the crates in a `Cargo.lock`. - Finding out who can publish a crate, users and GitHub teams both. - Release monitoring on a schedule, diffing `maxVersion` and `updatedAt` between runs. ## When not to use it - Crate source, README text or rendered documentation. You get registry metadata and the links that point at the full text. - Transitive dependency resolution. `dependencies` is the direct dependency list of one version, not a resolved graph. Feed the output back in to walk a level deeper. - Download time series. There is one all-time figure and one 90-day figure, no daily history. - Anything outside crates.io: no docs.rs build status, no GitHub activity, no advisories. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/crates-io-scraper').call({ crates: ['serde', 'tokio', 'clap'], includeVersions: false, includeDependencies: true, includeOwners: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~crates-io-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"crates":["serde","tokio"],"includeDependencies":true}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/crates-io-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `crates` | string[] | yes | | Crate names (`serde`), or crates.io, docs.rs and lib.rs URLs, which are normalised to names. Mixed forms are fine. Duplicates are removed. Hyphens and underscores are interchangeable when a name is resolved. | | `includeVersions` | boolean | no | `false` | Adds `versions`: every published version with its own downloads, licence, MSRV, edition, crate size, yank status and publisher. Costs no extra request, only record size. | | `includeDependencies` | boolean | no | `false` | Adds `dependencies`: the direct dependencies of the default version. One extra request per crate. | | `includeOwners` | boolean | no | `false` | Adds `owners`: the users and GitHub teams allowed to publish. One extra request per crate. | **`includeVersions` costs requests you do not pay; the other two cost requests you do.** The version list arrives on the same response as the crate itself, so switching it on never slows a run, it only inflates records: `serde` alone carries over 300 versions. `includeDependencies` and `includeOwners` each add one request per crate, so a 1000-crate run goes from 1000 requests to 3000, and crates are fetched sequentially at roughly one per second. Set the run timeout from the request count, not the crate count. ## Output One record per crate that resolved. | Field | Type | Notes | |---|---|---| | `name` | string | Canonical crate name as the registry stores it. Join on this, not on the string you passed. | | `description` | string \| null | One-line description from `Cargo.toml`. | | `maxVersion` | string \| null | Highest stable version, falling back to the highest version overall when no stable release exists. | | `downloads` | number | All-time downloads. | | `recentDownloads` | number | Downloads in the last 90 days. This is the adoption signal, not `downloads`. | | `categories` | string[] | Human-readable category names (`Encoding`, not the `encoding` slug). | | `keywords` | string[] | Author-supplied keywords. | | `license` | string \| null | SPDX expression, taken from the default version. See below. | | `repository` | string \| null | Source URL as declared by the author. | | `documentation` | string | Declared docs URL, falling back to the crate's docs.rs page. Never null, so never treat it as evidence the author set one. | | `versionCount` | number | Total published versions, yanked ones included. | | `createdAt` | string | First publish, ISO 8601. | | `updatedAt` | string | Most recent publish. The maintenance signal. | | `versions` | object[] | Only with `includeVersions`. Keys: `num`, `downloads`, `license`, `yanked`, `crateSize`, `rustVersion`, `edition`, `publishedBy`, `createdAt`. | | `dependencies` | object[] | Only with `includeDependencies`. Keys: `crate`, `req`, `kind` (`normal`, `dev` or `build`), `optional`, `defaultFeatures`, `features`, `target`. | | `owners` | object[] | Only with `includeOwners`. Keys: `login`, `name`, `kind` (`user` or `team`), `url`. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record with all three toggles on, arrays trimmed: ```json { "name": "serde", "description": "A generic serialization/deserialization framework", "maxVersion": "1.0.229", "downloads": 1238022395, "recentDownloads": 258045014, "categories": ["Encoding", "No standard library", "No dynamic allocation"], "keywords": ["no_std", "serde", "serialization"], "license": "MIT OR Apache-2.0", "repository": "https://github.com/serde-rs/serde", "documentation": "https://docs.rs/serde", "versionCount": 316, "createdAt": "2014-12-05T20:20:39.487502Z", "updatedAt": "2026-07-18T23:05:13.266456Z", "versions": [ { "num": "1.0.229", "downloads": 20270930, "license": "MIT OR Apache-2.0", "yanked": false, "crateSize": 83669, "rustVersion": "1.56", "edition": "2021", "publishedBy": "dtolnay", "createdAt": "2026-07-18T23:05:13.266456Z" } ], "dependencies": [ { "crate": "serde_core", "req": "=1.0.229", "kind": "normal", "optional": false, "defaultFeatures": false, "features": ["result"], "target": null } ], "owners": [ { "login": "dtolnay", "name": "David Tolnay", "kind": "user", "url": "https://github.com/dtolnay" }, { "login": "github:serde-rs:publish", "name": "publish", "kind": "team", "url": "https://github.com/serde-rs" } ], "scrapedAt": "2026-08-06T11:33:38.693Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is the only place a partially populated record admits that it is partial. ```json { "cratesRequested": 4, "cratesFailed": 1, "failures": [ { "crate": "this-crate-does-not-exist-zzz9", "error": "crate \"this-crate-does-not-exist-zzz9\" not found (404)" } ], "cratesSaved": 3, "filters": { "includeVersions": true, "includeDependencies": true, "includeOwners": true }, "finishedAt": "2026-08-06T11:33:45.691Z" } ``` `cratesFailed` counts only crates that produced no record at all, so `cratesSaved` plus `cratesFailed` equals `cratesRequested`. Entries in `failures` whose `error` starts with `owners:` or `dependencies:` are a different animal: that crate **was** saved, with an empty array in place of the missing section. Scan the prefixes before trusting an empty `owners` or `dependencies`. ## Behaviour to plan around - **`license` is a property of a version, not of a crate.** The registry has no crate-level licence field, so the Actor reports the default version's. A relicensed crate shows today's licence with no hint of the old one. Switch on `includeVersions` for the per-version history. - **An empty `owners` or `dependencies` can mean "call failed", not "none".** The prefixed entries in `failures` are the only way to tell the two apart. - **`versionCount` includes yanked versions.** It will be higher than the list of releases anyone recognises. Filter on `versions[].yanked` when you need live releases only. - **`maxVersion` can be a pre-release.** It is the highest stable version, but a crate that has never cut a stable release falls back to the highest version overall. - **`documentation` is never null.** When the author declared none, a docs.rs URL is synthesised from the crate name. That URL is a good guess, not a promise the page exists. - **The record does not echo your input.** A run given a URL or an underscore variant comes back keyed on the canonical `name`. Normalise your own side before joining. - **`dependencies` is one version deep, and only the default version.** No transitive resolution, and no history: a dependency added in an older release will not appear. - **A failing crate never aborts the run.** An unknown name is logged and recorded. The Actor only throws when every crate failed. - **Transient errors are retried** three times with linear backoff. Not-found responses are final for that crate. ## Recipes **Weekly ecosystem tracking.** All toggles off: one request per crate and the smallest record the Actor can produce. ```json { "crates": ["serde", "tokio", "clap", "reqwest", "rand", "regex"] } ``` Diff `recentDownloads`, `maxVersion` and `updatedAt` against the previous run, keyed on `name`. **Choose between competing crates.** Compare `recentDownloads` for current traction, `updatedAt` for maintenance, `license` for legal, and the length of `dependencies` for weight. ```json { "crates": ["reqwest", "ureq", "isahc", "hyper"], "includeDependencies": true } ``` **Supply-chain review of a Cargo.lock.** Feed in the crate names, take licence compatibility from `license` and publish rights from `owners`. ```json { "crates": ["tokio", "hyper", "openssl-sys", "libc"], "includeDependencies": true, "includeOwners": true } ``` Flag any crate where `owners` contains a single `user` entry: that is a one-person publish path into your build. **Release archaeology on one crate.** Versions are free in requests, so a single-crate run returns the whole publish history in seconds. ```json { "crates": ["serde"], "includeVersions": true } ``` Sort `versions` by `createdAt` for cadence, or sum `downloads` per major version to see where users actually sit.