--- name: zenodo-records-scraper description: Search Zenodo, CERN's open research repository, and return one structured record per deposit via the Apify Actor arman-bd/zenodo-records-scraper. Each record carries the record id and DOI, title, plain-text description, creators with affiliations and ORCIDs, resource type, publication date, licence, keywords, communities, file listings with sizes, checksums and direct download URLs, public download and view counts, version and related identifiers. Use when a task needs open dataset or research-software discovery, community or lab monitoring, corpus building for retrieval, or usage-based bibliometrics. Not for paywalled literature, citation counts, or file contents. --- # Zenodo Scraper: Datasets, Software & Research Output Apify Actor `arman-bd/zenodo-records-scraper`. Give it search terms, resource types, communities, or nothing at all, and it returns one dataset record per Zenodo deposit. Every filter combination runs as its own independent search. You supply no credentials. ## When to use it - Finding open datasets or research software on a topic, with direct file URLs. - Watching a community, a lab name or a keyword on a schedule for new deposits. - Building a retrieval corpus: clean plain-text abstracts with DOIs to cite. - Usage-based bibliometrics, since download and view counts are published per record. - Resolving a known DOI or author into full deposit metadata through the search syntax. ## When not to use it - Paywalled or commercially published literature. Zenodo holds what depositors upload. - Citation counts or impact factors. You get views and downloads, which are a different measure entirely. - File contents. You get names, sizes, checksums and URLs, never the bytes. - Exhaustive harvesting of a huge result set. A single search cannot be paged beyond 10,000 records; narrow the query instead. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/zenodo-records-scraper').call({ searchQueries: ['machine learning'], resourceTypes: ['dataset'], communities: [], includeFiles: true, maxResults: 500, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~zenodo-records-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchQueries":["protein structure"],"resourceTypes":["dataset"],"maxResults":200}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/zenodo-records-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input Every field is optional. With no filters at all you get the most recent deposits in the whole repository, up to `maxResults`. | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchQueries` | string[] | no | `[]` | Full-text terms, passed through to Zenodo's own search syntax, so fielded queries work: `creators.name:"Smith"`, `doi:"10.5281/zenodo.16537543"`. Quote a phrase to match it exactly. Each entry is a separate search, never AND-ed with the others. Empty means no text filter. | | `resourceTypes` | string[] | no | `[]` | Machine-readable types: `dataset`, `software`, `publication`, `poster`, `presentation`, `image`, `video`, `lesson`, `physicalobject`, `other`. Lower-cased for you. Each entry is a separate search. Empty means all types. | | `communities` | string[] | no | `[]` | Community slugs, or a full community page link, which is reduced to the slug. Each entry is a separate search. Empty means all of Zenodo. | | `includeFiles` | boolean | no | `true` | Include the file list with names, byte sizes, checksums and download URLs. Off removes the `files` key from every record. | | `maxResults` | integer | no | `500` | Total cap across all searches, split evenly between them, minimum `0`. `0` means no limit. | **Filters multiply, they do not narrow.** Repeating a facet in one request would make Zenodo honour only the last value, so this Actor runs the cross product instead: two queries with two types is four independent searches, each paginated and retried on its own, and one failing does not touch the others. That has two consequences worth planning for. `maxResults` is divided by the number of combinations, so adding a type quietly halves the depth of every existing one. And a record matching two combinations is saved twice, because nothing is de-duplicated across searches. ## Output One record per deposit, per matching search. | Field | Type | Notes | |---|---|---| | `recordId` | number \| null | Zenodo's record id. Use it, with `doi`, to de-duplicate. | | `doi` | string \| null | Minted DOI. Present on essentially every deposit. | | `title` | string \| null | Record title. | | `description` | string | Abstract as plain text: depositor HTML stripped and entities decoded, including accented names and unit symbols. Empty string when there is none, never null. | | `creators` | object[] | Each `{ name, affiliation, orcid }`. `affiliation` and `orcid` are frequently null. | | `resourceType` | string \| null | Machine-readable type, e.g. `dataset`. Round-trips straight back into `resourceTypes`. | | `publicationDate` | string \| null | `YYYY-MM-DD`. Depositor-declared, so it can predate the deposit. | | `license` | string \| null | Licence id in Zenodo's lower-case form, e.g. `cc-by-4.0`, `mit`. `null` when unset. | | `keywords` | string[] | Depositor-supplied, uncontrolled vocabulary. | | `communities` | string[] | Community slugs the record belongs to. | | `files` | object[] | Each `{ name, size, checksum, downloadUrl }`, `size` in bytes and `checksum` prefixed with its algorithm. **The key is absent entirely** when `includeFiles` is off. | | `downloads` | number \| null | Public download count. | | `views` | number \| null | Public view count. | | `version` | string \| null | Version string when the depositor set one. Free text, not semver. | | `relatedIdentifiers` | object[] | Each `{ identifier, relation, scheme }`, e.g. relation `isSupplementTo` scheme `doi`. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, description and file list trimmed: ```json { "recordId": 21805794, "doi": "10.5281/zenodo.21805794", "title": "Submerged Aquatic Vegetation (SAV) in the Balearic Islands", "description": "Dataset description\n\nAnnual map of submerged aquatic vegetation along the coast …", "creators": [ { "name": "König, Marcel", "affiliation": "Brockmann Consult (Germany)", "orcid": "0000-0002-7617-888X" } ], "resourceType": "dataset", "publicationDate": "2026-08-05", "license": "cc-by-4.0", "keywords": ["remote sensing", "Sentinel-2"], "communities": ["eu", "foccus-project"], "files": [ { "name": "UMPSupportingDoc_BAT_V1.pdf", "size": 947143, "checksum": "md5:9856d723eea345ec504d88a32d16d7f1", "downloadUrl": "https://zenodo.org/api/records/21805794/files/UMPSupportingDoc_BAT_V1.pdf/content" } ], "downloads": 0, "views": 105, "version": "1.0", "relatedIdentifiers": [ { "identifier": "10.5281/zenodo.17579480", "relation": "isDescribedBy", "scheme": "doi" } ], "scrapedAt": "2026-08-06T11:47:54.497Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** With the cross product in play it is the only way to see how many searches actually ran and which of them died. ```json { "sourcesRequested": 4, "sourcesFailed": 1, "failures": [ { "source": "q=\"climate model\" type=software", "error": "invalid query (400), …" } ], "recordsSaved": 1489, "filters": { "searchQueries": ["\"climate model\"", "genome assembly"], "resourceTypes": ["dataset", "software"], "communities": [], "includeFiles": false, "maxResults": 2000 }, "finishedAt": "2026-08-06T11:52:11.004Z" } ``` `sourcesRequested` is the size of the cross product, which is what your budget was divided by. Each `failures[].source` is the human-readable label of one combination, so you can see exactly which query and type pair failed rather than re-running everything. `recordsSaved` below `maxResults` is normal and usually means a combination exhausted its result set; it can also sit slightly *above*, because the per-search budget is rounded up. The run throws only when every combination failed. ## Behaviour to plan around - **Results come back newest first, always.** There is no sort input. A cap therefore gives you the most recent N, never the most relevant or the most downloaded. To rank by `downloads`, pull a generous set and sort client-side. - **The same record can appear twice.** Nothing de-duplicates across combinations, so a deposit matching two of your queries produces two dataset items. De-duplicate on `recordId` before counting anything. - **`maxResults` is a total, split evenly and rounded up.** Four combinations with a cap of 2000 means 500 each, and a combination that runs dry does not donate its remainder to the others. Set the cap with the number of combinations in mind. - **`files` is missing, not null, when `includeFiles` is off.** Code that reads the key without a guard sees `undefined`, and CSV exports lose the column. Do not test it for null. - **An empty `files` array means restricted access.** Closed and embargoed deposits publish their metadata but not their files. That is the registry's answer, not a scrape failure, and the DOI is still resolvable for a human. - **Depth stops at 10,000 records per search.** The Actor stops one page short of that boundary and logs a warning rather than letting the request be rejected. If you need more, split the query, by community, by type or by narrower terms. - **A rejected query is fatal for that combination only.** Bad search syntax comes back as an invalid-query failure carrying the source's own explanation, and the other combinations continue. - **Searches can be slow.** A filtered query routinely takes tens of seconds, so requests allow 90 s and server faults are retried four times with exponential backoff, honouring any wait the source asks for. Budget run time accordingly on a wide sweep. - **`keywords` is uncontrolled and `version` is free text.** Neither is normalised upstream, so match on them loosely or not at all. ## Recipes **Open datasets for a topic, ranked by use.** One combination, so the whole budget goes to it. ```json { "searchQueries": ["machine learning"], "resourceTypes": ["dataset"], "maxResults": 500 } ``` Sort the result by `downloads` yourself, then take `files[].downloadUrl` for the ones you want. Check `license` before using anything. **Everything a community has published.** File listings kept, since a community sweep is usually the prelude to fetching. ```json { "communities": ["biosyslit"], "includeFiles": true, "maxResults": 1000 } ``` Schedule it and diff on `recordId` to catch new deposits, since results arrive newest first. **Two topics across two types.** Four independent searches, budget split four ways. ```json { "searchQueries": ["\"climate model\"", "genome assembly"], "resourceTypes": ["dataset", "software"], "includeFiles": false, "maxResults": 2000 } ``` Expect up to 500 per combination and de-duplicate on `recordId`, because a record can match both queries. **Resolve a known DOI or author.** Fielded search, tiny cap. ```json { "searchQueries": ["doi:\"10.5281/zenodo.16537543\""], "maxResults": 5 } ``` The same pattern works for `creators.name:"Smith"`, which is the cheapest way to turn a name into DOIs and ORCIDs.