--- name: pubmed-articles-scraper description: Search PubMed and return one flat record per biomedical article via the Apify Actor arman-bd/pubmed-articles-scraper. Each row carries the PMID, DOI and PMC ID, title, full abstract with its structured section labels intact, ordered author list and distinct affiliations, journal title and ISO abbreviation, ISSN, volume, issue and pages, publication date and year, NLM MeSH terms, author keywords, publication types, language and journal country. Use when a task needs systematic-review screening, a biomedical RAG corpus, institutional output tracking, horizon scanning on a schedule, or reconciling PMIDs to DOIs. Not for full text, figures, citation or reference lists, citation counts, clinical trial registrations, or preprint servers. --- # PubMed Articles Scraper Apify Actor `arman-bd/pubmed-articles-scraper`. Give it PubMed query strings, a list of PMIDs, or both, and get one dataset record per article. It runs without credentials. An NCBI API key is optional and only raises the pace. ## When to use it - Systematic review screening: one query per PICO arm, narrowed to `Review` and `Meta-Analysis`, exported for triage. - Building a biomedical retrieval corpus, where `title` plus `abstract` plus `meshTerms` is a chunk that arrives with its own controlled vocabulary attached. - Institutional or group output tracking, using an `[Affiliation]` query and counting by `publicationYear`. - Horizon scanning on a schedule: `sortBy: "pub_date"` with a rolling `fromDate`, diffed on `pmid`. - Reconciling a list of PMIDs you already hold to DOIs and PMC IDs for joining onto other sources. ## When not to use it - Full text, figures, tables or supplementary material. You get the abstract, and a `pmcId` when a free full text exists elsewhere. - References, cited-by lists or citation counts. None are returned. - Clinical trial registrations, grant records or author identifiers such as ORCID. - Preprint servers. PubMed indexes some preprints, but this is not the tool for a preprint feed. - Exhaustive retrieval of a broad query. One query tops out at 9,999 records, so a sweep that large has to be sliced by date. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/pubmed-articles-scraper').call({ searchQueries: ['long covid[MeSH Terms]', 'post-acute sequelae SARS-CoV-2'], articleTypes: ['Review', 'Meta-Analysis'], language: 'eng', fromDate: '2024-01-01', sortBy: 'pub_date', includeAbstract: true, maxResults: 500, email: 'you@example.com', }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~pubmed-articles-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchQueries":["CRISPR gene editing"],"articleTypes":["Review"],"fromDate":"2025-01-01","maxResults":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/pubmed-articles-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input At least one of `searchQueries` or `pmids` is required. PubMed has no "return everything" mode. | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchQueries` | string[] | no | `[]` | PubMed query syntax, passed through untouched. Field tags work: `[Title]`, `[Author]`, `[Journal]`, `[MeSH Terms]`, `[Affiliation]`. Combine with `AND`, `OR`, `NOT`. Whitespace is collapsed and duplicates removed. | | `pmids` | string[] | no | `[]` | Bare PMID, `PMID: 33301246`, or a PubMed article URL. The first run of 4 to 9 digits is taken as the ID. | | `fromDate` | string | no | `""` | `YYYY-MM-DD`, `YYYY-MM` or `YYYY`. Applied as PubMed's own publication-date filter, not as text in the query. A bare year resolves to 1 January. Any other format throws. | | `toDate` | string | no | `""` | Same formats. A bare year resolves to 31 December, so `2024` means the whole year. Any other format throws. | | `articleTypes` | string[] | no | `[]` | Publication types, ORed together and ANDed onto every query. Values outside the known list are passed through with a warning rather than rejected. | | `language` | string | no | `""` | Three-letter code such as `eng`, `fre`, `ger`, `spa`. This is the language of publication; non-English papers usually still have an English abstract. | | `sortBy` | string | no | `"relevance"` | One of `relevance`, `pub_date`, `Author`, `JournalName`. Anything else throws. Decides which records survive the cap. | | `includeAbstract` | boolean | no | `true` | Off leaves `abstract` as `null` and makes the dataset several times smaller. | | `maxResults` | integer | no | `100` | Cap **per search query**, 1 to 9999. Does not apply to `pmids`. | | `email` | string | no | `""` | Passed to NCBI so it can contact you before blocking a heavy client. Never used by the Actor for anything else. | | `apiKey` | string | no | `""` | Free NCBI key. Raises the pace from roughly 3 requests per second to 10. Secret field. | **Batch economics decide everything here, and they favour big runs.** A query costs one search request, and the matching records are then pulled 200 at a time, so 1,000 articles is one search plus five fetches while 50 articles is one search plus one fetch. Cost per article falls sharply as `maxResults` rises, and `includeAbstract` changes only the response size, not the request count. The real constraint is the 9,999 ceiling on a single query: above that you must slice by `fromDate` and `toDate` and run several queries, not raise the cap. `sortBy` matters most exactly when you are capped, because it decides which 100 of 40,000 matches you keep: use `pub_date` for monitoring and `relevance` for exploration. ## Output One record per article, deduplicated on `pmid` across every query and the PMID list. | Field | Type | Notes | |---|---|---| | `pmid` | string \| null | PubMed ID, as a **string**. Join and dedup key. | | `doi` | string \| null | Publisher DOI, from the ID list or the article's location ID. | | `pmcId` | string \| null | PubMed Central ID. Present only when a free full text exists there. | | `title` | string \| null | Article title with inline presentational markup stripped. | | `abstract` | string \| null | Full abstract. Structured abstracts keep their headings inline. `null` when `includeAbstract` is off or the record genuinely has none. | | `authors` | string[] | Author names in order. Consortium authors appear under their collective name. | | `affiliations` | string[] | Every distinct affiliation string on the record, deduplicated. Not aligned positionally with `authors`. | | `journal` | string \| null | Full journal title, or the book title for Bookshelf records. | | `journalAbbrev` | string \| null | ISO abbreviation. | | `issn` | string \| null | Journal ISSN. | | `volume` | string \| null | As a string. Absent on book records. | | `issue` | string \| null | As a string. Absent on book records. | | `pages` | string \| null | Page range as printed, for example `415-431`. | | `publicationDate` | string \| null | The most precise date the record supports: `2020-12-31`, `2026-12` or `1929`. Not a fixed-width date. | | `publicationYear` | number \| null | The four-digit year pulled out of that date. Use this for grouping. | | `meshTerms` | string[] | NLM Medical Subject Headings assigned by indexers. | | `keywords` | string[] | Author-supplied keywords, deduplicated. | | `publicationTypes` | string[] | `Journal Article`, `Review`, `Randomized Controlled Trial`, `Published Erratum` and so on. | | `language` | string \| null | Publication language code. | | `country` | string \| null | The journal's country of origin, not the authors' country. | | `url` | string \| null | PubMed page for the article. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, abstract trimmed: ```json { "pmid": "42522380", "doi": "10.1080/21691401.2026.2702869", "pmcId": null, "title": "Recent advances of CRISPR-based gene editing technologies and delivery strategies.", "abstract": "CRISPR technology is a powerful tool for gene editing, in which the efficient delivery of living target cells allows it to show great clinical potential. …", "authors": ["Fanyu Xue", "Zixuan Xin", "Gaojie Wang", "Jiayin Xing", "Hengxin Han", "Xiangxiang Zhao", "Ningning Song"], "affiliations": ["College of Life Science and Technology, Shandong Second Medical University, Weifang, China."], "journal": "Artificial cells, nanomedicine, and biotechnology", "journalAbbrev": "Artif Cells Nanomed Biotechnol", "issn": "2169-141X", "volume": "54", "issue": "1", "pages": "415-431", "publicationDate": "2026-12", "publicationYear": 2026, "meshTerms": ["Humans", "Gene Editing", "Animals", "CRISPR-Cas Systems", "Gene Transfer Techniques"], "keywords": ["Adeno-associated viral vectors", "CRISPR-Cas", "LNP delivery systems", "delivery system"], "publicationTypes": ["Journal Article", "Review"], "language": "eng", "country": "England", "url": "https://pubmed.ncbi.nlm.nih.gov/42522380/", "scrapedAt": "2026-08-06T12:15:43.691Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "sourcesRequested": 2, "sourcesFailed": 0, "failures": [], "searchesRun": 2, "pmidsRequested": 0, "articlesSaved": 486, "usedApiKey": false, "filters": { "searchQueries": ["long covid[MeSH Terms]", "post-acute sequelae SARS-CoV-2"], "pmids": [], "fromDate": "2024/01/01", "toDate": null, "articleTypes": ["Review", "Meta-Analysis"], "language": "eng", "sortBy": "pub_date", "includeAbstract": true, "maxResults": 500 }, "finishedAt": "2026-08-06T12:16:02.118Z" } ``` A *source* is one search query, or the whole PMID list treated as a single unit. Any non-empty `failures` means a query contributed nothing, and each entry names the `source` label and the `error`. Note the date fields in `filters` are shown in the form actually sent, not the form you typed, so `2024` comes back as `2024/01/01` or `2024/12/31` depending on which end it was. `articlesSaved` below the sum of your caps is normal: deduplication across overlapping queries, and unknown PMIDs, both reduce it without any failure being recorded. ## Behaviour to plan around - **`maxResults` does not apply to `pmids`.** It caps each search query only. A run with 4,000 PMIDs and `maxResults: 100` fetches all 4,000. - **Unknown PMIDs vanish without an error.** The source returns an empty set for an ID that does not exist, so a missing article shows up as a lower `articlesSaved` and nothing in `failures`. Reconcile your input list against the returned `pmid` values yourself. - **9,999 is a hard ceiling per query.** Asking for more is clamped. To sweep a broad topic, run the same query several times over successive `fromDate` and `toDate` windows; deduplication on `pmid` makes overlapping windows safe. - **`meshTerms` is empty on recent papers.** Indexers assign headings weeks to months after publication, so a paper from last week has `keywords` but no MeSH. Never treat an empty `meshTerms` as "not about this topic". - **`publicationDate` is variable precision.** `2020-12-31`, `2026-12` and `1929` are all valid values of the same field. Sort and group on `publicationYear`, and parse `publicationDate` defensively. - **Book records come back with fewer fields.** Bookshelf entries such as GeneReviews are parsed rather than dropped, but have no `volume`, no `issue` and no `meshTerms`, and `journal` holds the book title. - **`affiliations` is a deduplicated set, not per author.** There is no positional correspondence with `authors`, so you cannot say which author sits where. - **`articleTypes` is ORed, then ANDed onto every query.** Selecting `Review` and `Meta-Analysis` returns records that are either, and applies that constraint to all of your queries, not just the first. - **One failing query never aborts the run.** The Actor throws only when every source failed. A malformed query is rejected immediately rather than retried, while transient conditions get four attempts with backoff. - **`abstract` can be `null` even with `includeAbstract: true`.** Editorials, errata, letters and many older papers simply have none. Nothing is synthesised. ## Recipes **Systematic review screening.** One query per arm, narrowed by type and language, and a cap high enough that the sort order stops mattering. ```json { "searchQueries": [ "(semaglutide[Title/Abstract]) AND obesity", "(tirzepatide[Title/Abstract]) AND obesity" ], "articleTypes": ["Review", "Systematic Review", "Meta-Analysis"], "language": "eng", "fromDate": "2020", "maxResults": 2000, "email": "you@example.com" } ``` Export `pmid`, `title`, `abstract` and `publicationTypes` for triage; the run has already deduplicated across both arms. **Weekly horizon scan.** Newest first, a rolling window, abstracts on so you can rank what came in. ```json { "searchQueries": ["antimicrobial resistance[MeSH Terms]"], "fromDate": "2026-08-01", "sortBy": "pub_date", "maxResults": 500, "email": "you@example.com" } ``` Diff on `pmid` against the previous run. Move `fromDate` forward each time. **PMID reconciliation, metadata only.** Abstracts off keeps the dataset small when all you want is identifiers. ```json { "pmids": ["33301246", "32109013", "https://pubmed.ncbi.nlm.nih.gov/31978945/"], "includeAbstract": false } ``` Join your own table on `pmid` and take `doi` and `pmcId`. Rows absent from the result are IDs that do not resolve. **Rule of thumb for the API key.** Under a few thousand articles you will not notice the difference, because the run is dominated by one search plus a handful of 200-record fetches. Above roughly ten thousand articles across many queries, supply `apiKey` and `email` together.