--- name: go-module-scraper description: Resolve Go modules against the official Go module proxy as structured records via the Apify Actor arman-bd/go-module-scraper. Returns, per module, the version the go command would pick right now, its publish timestamp, the count and full sorted list of every published version, the go directive and the complete require list parsed from go.mod, plus whether the module carries a formal deprecation notice. Use for Go dependency audits, staleness and deprecation sweeps, release-cadence analysis or building a module catalogue. Not for source code, package documentation, go.sum checksums, private or GOPRIVATE modules, or standard-library packages. --- # Go Module Scraper Apify Actor `arman-bd/go-module-scraper`. Give it module import paths, get one dataset record per module, answered by the same service the `go` command itself consults rather than by a scrape of a package page. It runs without credentials; you supply nothing but the list. ## When to use it - Auditing the `require` lines of one or many `go.mod` files: what is stale, what is deprecated, what the transitive surface looks like. - A nightly staleness check, diffing `latestVersion` and `isDeprecated` per module. - Release cadence: the full version list with timestamps for the newest one, across a set of modules. - Confirming what `go get ` would resolve to today, including for modules that were never tagged. - Building a catalogue or explorer over a curated set of Go modules. ## When not to use it - Source code, README text, package documentation, symbols or examples. Only module metadata is returned. - `go.sum` checksum lines. Version identity and the resolved upstream repository URL are returned; the signed hash log is a separate service and is not read. - Private modules, or anything a team keeps behind `GOPRIVATE`. The public proxy cannot see them and they come back as failures. - Standard-library packages (`fmt`, `net/http`). They are not modules and are dropped from the input before any request. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/go-module-scraper').call({ modules: ['github.com/gorilla/mux', 'github.com/golang/protobuf', 'golang.org/x/net'], includeGoMod: true, includeAllVersions: false, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~go-module-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"modules":["github.com/gorilla/mux","github.com/gin-gonic/gin"],"includeGoMod":true,"includeAllVersions":false}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/go-module-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `modules` | string[] | yes | | Full import paths as they appear in a `go.mod` require line, host first: `github.com/gorilla/mux`, `golang.org/x/net`, `github.com/Masterminds/semver/v3`. A pkg.go.dev URL, an `https://` prefix, a query or fragment, and a trailing `@v1.2.3` are all stripped for you. Uppercase letters are proxy-encoded for you. Duplicates are removed. | | `includeGoMod` | boolean | no | `true` | Fetch and parse the `go.mod` of the resolved latest version. This is what populates `goVersion`, `requires`, `isDeprecated` and `deprecationNotice`; with it off, all four are `null`. Costs one extra request per module. | | `includeAllVersions` | boolean | no | `true` | Write the full sorted `versions` array into each row. `versionCount` is populated either way. Costs no extra request. | **Only one of the two booleans costs a request.** `includeGoMod` adds a fetch per module and is the entire point of a dependency audit, so leave it on for that. `includeAllVersions` changes row size, not run cost: a module like `golang.org/x/net` has hundreds of versions, so turning it off shrinks a large sweep considerably while still leaving `versionCount` to tell you how many there were. Off for breadth, on when you actually intend to compute release intervals. ## Output One record per module that resolved. | Field | Type | Notes | |---|---|---| | `module` | string | The module path after normalisation, which is what you should join back on. | | `encodedPath` | string | The proxy-encoded form actually requested: every uppercase letter written as `!` plus its lowercase. Identical to `module` for all-lowercase paths. | | `latestVersion` | string | What `go get ` would resolve to right now. A pseudo-version for untagged modules. | | `publishedAt` | string \| null | Commit timestamp of `latestVersion`, ISO 8601. | | `versionCount` | number | How many versions the version listing returned. `0` is legitimate, see below. | | `versions` | string[] | Every version, newest first, sorted by semver precedence. **Absent from the record entirely** when `includeAllVersions` is off, not `null`. | | `goVersion` | string \| null | The `go` directive from `go.mod`, the minimum toolchain the module claims. `null` when `includeGoMod` is off. | | `requires` | object[] \| null | Every requirement as `{ module, version, indirect }`, direct and indirect together. `null` when `includeGoMod` is off. | | `isDeprecated` | boolean \| null | Whether `go.mod` carries a `// Deprecated:` comment before its `module` directive. `null` when `includeGoMod` is off, which is not the same claim as `false`. | | `deprecationNotice` | string \| null | The text of that comment, usually naming the replacement module. | | `repoUrl` | string \| null | Upstream repository URL the proxy resolved the module from. | | `docsUrl` | string | Constructed pkg.go.dev URL for the module. Always present, never verified. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, the version list trimmed: ```json { "module": "github.com/gin-gonic/gin", "encodedPath": "github.com/gin-gonic/gin", "latestVersion": "v1.10.0", "publishedAt": "2024-05-07T09:12:18Z", "versionCount": 47, "versions": ["v1.10.0", "v1.9.1", "v1.9.0", "v1.8.2", "…"], "goVersion": "1.20", "requires": [ { "module": "github.com/bytedance/sonic", "version": "v1.11.6", "indirect": false }, { "module": "github.com/gin-contrib/sse", "version": "v0.1.0", "indirect": false }, { "module": "github.com/cloudwego/base64x", "version": "v0.1.4", "indirect": true } ], "isDeprecated": false, "deprecationNotice": null, "repoUrl": "https://github.com/gin-gonic/gin", "docsUrl": "https://pkg.go.dev/github.com/gin-gonic/gin", "scrapedAt": "2026-08-06T12:00:00.000Z" } ``` A deprecated module carries the marker in its `go.mod`: ```json { "module": "github.com/golang/protobuf", "latestVersion": "v1.5.4", "publishedAt": "2024-03-06T06:45:40Z", "isDeprecated": true, "deprecationNotice": "Use the \"google.golang.org/protobuf\" module instead." } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "modulesRequested": 4, "modulesSaved": 3, "modulesFailed": 1, "failures": [ { "module": "github.com/nonexistent/nope-xyz", "error": "not found (404): not found: module github.com/nonexistent/nope-xyz …" } ], "filters": { "includeGoMod": true, "includeAllVersions": false }, "finishedAt": "2026-08-06T12:00:03.402Z" } ``` `modulesRequested` counts paths that survived normalisation and deduplication, not the length of your input array. So `modulesSaved` plus `modulesFailed` always equals `modulesRequested`, and an entry you passed that appears in neither was discarded before the run started: it was a duplicate, or it had no slash and so was not a module path at all. Compare `modulesRequested` against your own input length to catch that. ## Behaviour to plan around - **`versions` is absent, not null, when `includeAllVersions` is off.** The key is not written at all. Code that reads `record.versions.length` throws rather than returning zero. Use `versionCount` when you only need the number. - **`null` in `goVersion`, `requires`, `isDeprecated` means not looked at.** With `includeGoMod` off, `isDeprecated: null` is not a claim that the module is healthy. Filter on `isDeprecated === true`, never on falsiness. - **`versionCount: 0` alongside a real `latestVersion` is normal.** The module was never tagged, so there is nothing to list, but the latest-version lookup still resolves a pseudo-version from the newest commit. That is logged as a warning and the row is still saved. - **The version list arrives unordered and is sorted here** by semver precedence, newest first: prereleases rank below their release, and Go pseudo-versions order correctly because their first prerelease identifier is a numeric timestamp. Do not re-sort lexically. - **From v2 onward the major version is part of the module path.** `github.com/pelletier/go-toml` and `github.com/pelletier/go-toml/v2` are two different modules with two separate version histories and two separate rows. Omitting the suffix silently gives you the v1 line. - **Uppercase letters are encoded before the request.** The proxy protocol requires every capital to be written as `!` plus its lowercase form so a case-insensitive file system cannot conflate two modules. You pass the normal path; `encodedPath` shows what went out. This is the usual cause of a spurious not-found when people query by hand. - **An entry with no slash is dropped silently.** Standard-library packages and typos never reach a request and are not in `failures`. If every entry is dropped, the Actor throws with an explanation. - **One bad module never aborts the run.** A typo or a private repository is recorded in `RUN_SUMMARY.failures` and the run continues. The Actor only throws when every module failed. - **Transient errors are retried** three times with linear backoff. Not-found and gone responses are final for that module and are not retried, because the answer will not change. - **`requires` mixes direct and indirect.** Split on the `indirect` flag before counting a dependency surface; the indirect entries are the flattened transitive closure the toolchain recorded, not first-order dependencies. ## Recipes **Dependency audit across repositories.** Feed every module in your `go.mod` files. ```json { "modules": [ "github.com/gin-gonic/gin", "github.com/golang/protobuf", "golang.org/x/net", "gopkg.in/yaml.v2" ], "includeGoMod": true, "includeAllVersions": false } ``` Flag every row with `isDeprecated === true` and read `deprecationNotice` for the replacement. Compare `latestVersion` against the version pinned in your own `go.mod`. **Release cadence.** The full history, no `go.mod` fetch. ```json { "modules": ["github.com/gorilla/mux", "github.com/pelletier/go-toml/v2"], "includeGoMod": false, "includeAllVersions": true } ``` `versions` is already newest first. Drop entries containing a `-` to keep stable releases only, then take differences between adjacent versions once you have dates for them. **Toolchain floor across a catalogue.** Which modules demand a recent Go. ```json { "modules": ["github.com/spf13/cobra", "github.com/stretchr/testify", "github.com/Masterminds/semver/v3"], "includeGoMod": true, "includeAllVersions": false } ``` Sort on `goVersion` to find the highest floor in the set; that is the minimum toolchain your build needs. Rows with `goVersion: null` predate the directive, they do not mean zero. **Nightly staleness feed.** Same list every night, small rows. ```json { "modules": ["github.com/gorilla/mux", "github.com/gin-gonic/gin"], "includeGoMod": true, "includeAllVersions": false } ``` Diff on `module` plus `latestVersion` against the previous run; a change is a new release, and a flip in `isDeprecated` is worth an alert on its own.