--- name: rubygems-scraper description: Turn a list of Ruby gem names into flat structured records via the Apify Actor arman-bd/rubygems-scraper. Each record carries the latest version and platform, all-time and per-version download counts, the release date, authors, description, licences, yank status, checksum, every declared link (homepage, source, docs, changelog, bug tracker, mailing list, wiki, funding), the raw gemspec metadata map, runtime and development dependencies, and optional full version history. Use when a task needs a Gemfile licence or dependency audit, adoption and upgrade-velocity tracking, abandonment detection, or a monitor for newly published gems. Not for gem source code, private or self-hosted gem servers, or non-Ruby ecosystems. --- # RubyGems Scraper Apify Actor `arman-bd/rubygems-scraper`. Give it gem names, get one dataset record each. With no names at all it returns the most recently published gems, which makes an empty input a ready-made release monitor. No credentials are involved. ## When to use it - A licence and dependency audit over everything in a `Gemfile.lock`. - Adoption tracking: `versionDownloads` against `downloads` is a usable proxy for how fast a new release is being taken up. - Abandonment detection: a high `releaseCount` next to an old `lastRelease`. - Link harvesting: changelog, source, bug tracker and funding URLs for a set of gems, in one pass. - Release monitoring on a schedule, using the empty-input mode to catch gems you were not already tracking. ## When not to use it - Private or self-hosted gem servers. Public RubyGems only. - Gem source code, file listings or diffs. Metadata and links only. - Resolving a dependency tree. You get each gem's declared requirements for its latest version, not a solved lock file, and not historical dependency graphs. - Owner, ownership-change or download-by-day time series. - npm, PyPI, Packagist or any non-Ruby ecosystem. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/rubygems-scraper').call({ packages: ['rails', 'puma', 'nokogiri', 'devise'], includeVersionHistory: true, maxVersions: 50, includeDependencies: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~rubygems-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"packages":["rails","sidekiq"],"includeVersionHistory":true,"maxVersions":10}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/rubygems-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `packages` | string[] | no | `[]` | Gem names, or a RubyGems gem URL, which is reduced to the name. A trailing `.json`, surrounding quotes and anything after a version operator are stripped. Blanks and duplicates are dropped. **Case is preserved and it matters.** Empty falls back to the recently-published feed. | | `includeVersionHistory` | boolean | no | `false` | Add the `releases` array and fill in `releaseCount` and `firstRelease`. One extra request per gem, and a large response for gems with hundreds of versions. | | `maxVersions` | integer | no | `20` | Entries kept in `releases`, counting back from the newest. Ignored when `includeVersionHistory` is off. `releaseCount` still reports the true total. Maximum 2000. | | `includeDependencies` | boolean | no | `true` | Include `runtimeDependencies` and `developmentDependencies`. They arrive with the gem lookup, so this costs nothing; turn it off only to slim the dataset. | | `maxPackages` | integer | no | `25` | Cap on gems per run. Applies to your list and to the fallback feed alike. Maximum 5000. | **`includeVersionHistory` is the only switch that costs anything**, and it doubles the request count. Everything on the record's first two thirds (version, downloads, licences, links, metadata, dependencies) comes from a single request per gem, so a broad sweep of hundreds of gems is cheap as long as history is off. Turn it on for a short list where you need per-version dates, checksums or Ruby requirements, and keep `maxVersions` low: the count is reported truthfully either way, so a small cap costs you detail rather than accuracy. ## Output One record per gem that resolved, de-duplicated on the name RubyGems returned. | Field | Type | Notes | |---|---|---| | `name` | string | Gem name as RubyGems spells it, which may differ in case from your input. | | `version` | string \| null | Latest published version. | | `platform` | string \| null | `ruby` for pure Ruby, otherwise a native variant such as `java` or `x86_64-linux`. | | `lastRelease` | string \| null | When the latest version was pushed, ISO 8601. | | `downloads` | number \| null | All-time downloads across every version ever published. | | `versionDownloads` | number \| null | Downloads of the latest version alone. | | `authors` | string \| null | A single free-text string, not a list. Do not split it on commas: names such as `Datadog, Inc.` contain one. | | `info` | string \| null | The gem's description, verbatim from the gemspec. | | `licenses` | string[] | Declared SPDX identifiers. Empty array when nothing is declared. | | `yanked` | boolean | True when the latest version was withdrawn. | | `sha` | string \| null | Checksum of the latest version. | | `projectUri` | string \| null | RubyGems page for the gem. | | `gemUri` | string \| null | Direct download URL for the packaged gem. | | `homepageUri` | string \| null | Declared homepage. | | `documentationUri` | string \| null | Declared docs link. | | `sourceCodeUri` | string \| null | Declared source repository. | | `bugTrackerUri` | string \| null | Declared issue tracker. | | `changelogUri` | string \| null | Declared changelog. | | `mailingListUri` | string \| null | Declared mailing list. | | `wikiUri` | string \| null | Declared wiki. | | `fundingUri` | string \| null | Declared funding link. | | `metadata` | object | The raw gemspec metadata map, including flags such as `rubygems_mfa_required`. Empty object when unset. | | `runtimeDependencies` | object[] | Present only when `includeDependencies` is on. Each entry `{ name, requirements }`, for the latest version only. | | `developmentDependencies` | object[] | Present only when `includeDependencies` is on. Same shape. | | `releaseCount` | number \| null | True total of published versions. `null` unless version history was fetched. | | `firstRelease` | string \| null | Release date of the oldest version. `null` unless version history was fetched. | | `releases` | object[] | Present only when version history was fetched. Newest first, keys `version`, `platform`, `releasedAt`, `downloads`, `prerelease`, `rubyVersion`, `licenses`, `sha`. | | `scrapedAt` | string | Run timestamp, ISO 8601 UTC. | A real record, history trimmed to one entry: ```json { "name": "rails", "version": "8.1.3.1", "platform": "ruby", "lastRelease": "2026-07-29T15:02:41.060Z", "downloads": 772174102, "versionDownloads": 981377, "authors": "David Heinemeier Hansson", "info": "Ruby on Rails is a full-stack web framework optimized for programmer happiness and sustainable productivity.", "licenses": ["MIT"], "yanked": false, "sha": "ccd11a36bfc171bf9c66d585d14c0ece91c0c9dde840aae60c0118d6f5c9c52a", "projectUri": "https://rubygems.org/gems/rails", "gemUri": "https://rubygems.org/gems/rails-8.1.3.1.gem", "homepageUri": "https://rubyonrails.org", "documentationUri": "https://api.rubyonrails.org/v8.1.3.1/", "sourceCodeUri": "https://github.com/rails/rails/tree/v8.1.3.1", "bugTrackerUri": "https://github.com/rails/rails/issues", "changelogUri": "https://github.com/rails/rails/releases/tag/v8.1.3.1", "mailingListUri": "https://discuss.rubyonrails.org/c/rubyonrails-talk", "wikiUri": null, "fundingUri": null, "metadata": { "rubygems_mfa_required": "true" }, "runtimeDependencies": [ { "name": "activesupport", "requirements": "= 8.1.3.1" }, { "name": "bundler", "requirements": ">= 1.15.0" } ], "developmentDependencies": [], "releaseCount": 519, "firstRelease": "2009-07-25T18:01:56.000Z", "releases": [ { "version": "8.1.3.1", "platform": "ruby", "releasedAt": "2026-07-29T15:02:41.060Z", "downloads": 984407, "prerelease": false, "rubyVersion": ">= 3.2.0", "licenses": ["MIT"], "sha": "ccd11a36bfc171bf9c66d585d14c0ece91c0c9dde840aae60c0118d6f5c9c52a" } ], "scrapedAt": "2026-08-06T11:41:07.204Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "packagesRequested": 4, "packagesSaved": 3, "packagesFailed": 1, "requestsMade": 7, "failures": [ { "package": "redcloth", "error": "gem not found on RubyGems (404)" }, { "package": "nokogiri", "error": "version history: rate limited (429)" } ], "unknownPackages": ["redcloth"], "packageSource": "input", "filters": { "includeVersionHistory": true, "maxVersions": 50, "includeDependencies": true, "maxPackages": 25 }, "finishedAt": "2026-08-06T11:41:08.100Z" } ``` `packagesFailed` counts only names that did not exist, so it is not the length of `failures`: a gem whose version-history request failed appears in `failures` and is still saved, with `releaseCount` and `firstRelease` `null` and no `releases` key. `packagesSaved` can also fall short of `packagesRequested` with nothing in `failures`, which means two of your inputs resolved to the same gem and the duplicate was skipped. `packageSource` is `input` or `just-updated-feed`, which is how you catch an input array that was empty when you did not mean it to be. ## Behaviour to plan around - **Gem names are case-sensitive.** `RedCloth` resolves and `redcloth` does not. A mixed-case name that misses is retried once in lowercase, so `Rails` still finds `rails`, but the reverse cannot be recovered: nothing about an all-lowercase name says where the capitals belong. Paste names exactly as the gem's page spells them. - **`name` in the output is RubyGems' spelling, not yours.** After a lowercase retry the record carries the resolved name. Join on it, not on your input string. - **An empty `packages` array does not fail, it changes the job.** The run returns the most recently published gems instead. That feed tops out at 50 entries, so `maxPackages` above 50 does not widen it. Check `packageSource`. - **`runtimeDependencies`, `developmentDependencies` and `releases` are absent, not null, when not requested.** The keys are missing from the record entirely. Test for presence, not for `null`. - **`releaseCount` and `firstRelease` are `null` without version history**, which is not the same as a gem having no history. Turn `includeVersionHistory` on before reading either as a fact. - **Version history can fail on its own.** The gem is still saved; only `failures` records the miss. Do not read a `null` `releaseCount` as "one release". - **`authors` is one string.** Splitting on commas invents authors that do not exist. - **Dependencies describe the latest version only.** History entries carry version, platform, date, downloads, Ruby requirement, licences and checksum, never requirements. - **Each platform variant is its own entry in `releases`.** A gem with native extensions publishes the same version number several times, once per `platform`, so counting distinct version numbers is not the same as counting entries. - **Requests are deliberately spaced** and made one gem at a time. Transient failures are retried three times, waiting longer when the source asks for it; a missing gem is final and is not retried. - **The run only throws when nothing was saved and every lookup failed.** ## Recipes **Licence and dependency audit of a Gemfile.** History off, so it is one request per gem. ```json { "packages": ["rails", "puma", "nokogiri", "devise", "sidekiq"], "includeDependencies": true, "includeVersionHistory": false, "maxPackages": 500 } ``` Group by `licenses` for the legal view, and recurse over the `name` values in `runtimeDependencies` in a second run to walk one level deeper. **Abandonment sweep.** History on, but shallow. ```json { "packages": ["rails", "rspec", "sidekiq"], "includeVersionHistory": true, "maxVersions": 5 } ``` Rank by the age of `lastRelease` against `releaseCount`. A gem with hundreds of releases and none in two years is the pattern worth flagging. `yanked` on the latest version is a separate, sharper signal. **Upgrade-velocity tracking.** Cheap enough to schedule daily. ```json { "packages": ["rails", "puma"], "includeVersionHistory": false, "includeDependencies": false } ``` Store `version`, `downloads` and `versionDownloads` per run. The ratio of `versionDownloads` to `downloads`, tracked over the days after a release, is the adoption curve. **New-release monitor.** No input at all. ```json { "maxPackages": 50 } ``` One request returns the most recently published gems. Diff on `name` plus `version` against the previous run to get just what changed.