--- name: semantic-scholar-scraper description: Search Semantic Scholar or resolve specific papers as flat structured records via the Apify Actor arman-bd/semantic-scholar-scraper. Returns one record per paper with the Semantic Scholar and corpus ids, DOI, arXiv, PubMed, PubMed Central and DBLP cross-references, title, abstract, the model-written TLDR, year and publication date, venue and journal placement, author names with their ids, citation, influential-citation and reference counts, fields of study, publication types, open-access PDF link, status and licence, and optionally the papers citing it. Use for systematic reviews, bibliometrics, identifier reconciliation, open-access full-text collection or building a retrieval corpus. Not for full text, reference lists, author profiles, field query syntax or boolean search operators. --- # Semantic Scholar Scraper Apify Actor `arman-bd/semantic-scholar-scraper`. Give it free-text searches, a list of paper identifiers, or both in one run, and get one flat dataset record per paper. Results are deduplicated by paper id across everything the run touched. It runs without credentials, though supplying a free API key makes a large run substantially faster. ## When to use it - Systematic reviews: sweep a topic, bound it by year and a citation floor, export for screening. - Identifier reconciliation: hand it DOIs from another system and get arXiv, PubMed, PubMed Central and DBLP identifiers for the same works. - Bibliometrics over time: re-run a fixed paper list and watch `citationCount` and `influentialCitationCount` move. - Retrieval corpora: title, abstract, TLDR and fields of study is a chunk with metadata already attached. - Full-text collection: with `openAccessOnly` on, every row carries a directly downloadable PDF link. ## When not to use it - Full text. The abstract and a PDF link are as far as it goes. - Reference lists or the citation graph in the outward direction. You get `referenceCount`, a number, and optionally the papers citing this one, never the papers it cites. - Author profiles, h-indexes or institution records. Author names and ids only. - Boolean or fielded query syntax. Searches are plain keywords, ranked by relevance. - Exhaustive retrieval of a large topic. One search stops at 999 papers. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/semantic-scholar-scraper').call({ searchQueries: ['retrieval augmented generation'], paperIds: [], fields: ['abstract', 'tldr', 'authors', 'openAccessPdf'], year: '2022-2026', minCitationCount: 25, openAccessOnly: true, maxResults: 200, apiKey: process.env.S2_API_KEY, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~semantic-scholar-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchQueries":["retrieval augmented generation"],"year":"2023-2026","minCitationCount":20,"maxResults":50}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/semantic-scholar-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input At least one of `searchQueries` or `paperIds` must be non-empty; there is no "everything" mode. The two combine freely in a single run. | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchQueries` | string[] | one of two | `[]` | Plain keyword searches, one per entry, ranked by relevance. No field syntax, no boolean operators. Whitespace is collapsed and duplicates removed. Each query paginates independently. | | `paperIds` | string[] | one of two | `[]` | Specific papers. Accepts a 40-character Semantic Scholar hash, a bare DOI starting with the `10.` registrant, a prefixed DOI, an arXiv identifier, a PubMed identifier, or a full URL from any of those services, which is stripped down to the identifier. | | `fields` | string[] | no | all seven | Which optional blocks to request: `abstract`, `tldr`, `authors`, `openAccessPdf`, `fieldsOfStudy`, `publicationTypes`, `journal`. Core metadata is always fetched. Unrecognised entries are warned about and ignored, not fatal. | | `includeCitations` | boolean | no | `false` | Add a `citations` array listing the papers citing each record. The payload grows fast, so leave it off unless you are building a citation graph. | | `year` | string | no | `""` | Restricts **searches only**. A single year, a closed range, or an open one in either direction. A malformed value aborts the run before any request. | | `minCitationCount` | integer | no | `0` | Drops **search hits** below this citation count. Ignored for direct lookups. | | `openAccessOnly` | boolean | no | `false` | Keeps only **search hits** the source has a downloadable PDF for. Ignored for direct lookups. | | `maxResults` | integer | no | `100` | Papers saved per search query. Silently clamped to 999, which is the depth relevance search reaches. Direct lookups are unaffected. | | `apiKey` | string | no | `""` | A free Semantic Scholar API key. Strongly recommended: it gives the run its own request allowance, and the Actor paces itself roughly three times faster with one. Without a key, a busy moment can leave a query in `failures` after its retries are spent. Secret field. | **The three search filters do not touch `paperIds`, and that asymmetry catches people out.** `year`, `minCitationCount` and `openAccessOnly` are applied by the source while searching; a paper you name explicitly is fetched and saved whatever its year, citations or availability. So a run mixing both kinds of input produces rows that do not all satisfy the filters. If you need the filters applied uniformly, run the lookups separately and filter the dataset yourself. ## Output One record per distinct paper, deduplicated on `paperId` across all searches and lookups. | Field | Type | Notes | |---|---|---| | `paperId` | string \| null | Semantic Scholar's 40-character identifier. The deduplication key. | | `corpusId` | number \| null | Numeric corpus identifier. | | `doi` | string \| null | Bare DOI, no prefix and no URL. | | `arxivId` | string \| null | arXiv identifier without a version suffix. | | `pubmedId` | string \| null | PubMed identifier. | | `pubmedCentralId` | string \| null | PubMed Central identifier. | | `dblpId` | string \| null | DBLP key. | | `title` | string \| null | Paper title. | | `abstract` | string \| null | Full abstract. Requires `abstract` in `fields`. | | `tldr` | string \| null | One-sentence model-written summary. Requires `tldr` in `fields`. Best covered in computer science, thinner elsewhere. | | `year` | number \| null | Publication year. | | `publicationDate` | string \| null | Exact date where known. `null` for a large share of conference papers, which have only a year. | | `venue` | string \| null | Venue or conference name. | | `journalName` | string \| null | Requires `journal` in `fields`. | | `journalVolume` | string \| null | Requires `journal` in `fields`. | | `journalPages` | string \| null | Requires `journal` in `fields`. Often present when `journalName` is not. | | `authors` | string[] | Author names in order. Requires `authors` in `fields`, otherwise an empty array. | | `authorIds` | string[] | Semantic Scholar author ids, positionally aligned with `authors` only when every author has one. | | `citationCount` | number \| null | Citations to this paper. | | `influentialCitationCount` | number \| null | The subset the source's classifier considers load-bearing rather than incidental. Typically a small fraction of the total. | | `referenceCount` | number \| null | How many works this paper cites. The works themselves are not returned. | | `fieldsOfStudy` | string[] | Subject classification. Requires `fieldsOfStudy` in `fields`. | | `publicationTypes` | string[] | Record type, for example `JournalArticle`, `Conference`, `Review`. Requires `publicationTypes` in `fields`. | | `isOpenAccess` | boolean \| null | Core metadata, always fetched, independent of the `openAccessPdf` field. | | `openAccessPdfUrl` | string \| null | Direct PDF link. Requires `openAccessPdf` in `fields`. | | `openAccessStatus` | string \| null | Open-access colour, upper case. Requires `openAccessPdf` in `fields`. | | `openAccessLicense` | string \| null | Licence code. Requires `openAccessPdf` in `fields`. | | `citations` | object[] \| null | `null` unless `includeCitations` is on. Keys: `paperId`, `title`, `year`, `citationCount`. | | `url` | string \| null | The paper's page on Semantic Scholar. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, fetched by DOI, long strings trimmed: ```json { "paperId": "df2b0e26d0599ce3e70df8a9da02e51594e0e992", "corpusId": 52967399, "doi": "10.18653/v1/N19-1423", "arxivId": "1810.04805", "pubmedId": null, "pubmedCentralId": null, "dblpId": "journals/corr/abs-1810-04805", "title": "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", "abstract": "We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers…", "tldr": "A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text…", "year": 2019, "publicationDate": null, "venue": "North American Chapter of the Association for Computational Linguistics", "journalName": null, "journalVolume": null, "journalPages": "4171-4186", "authors": ["Jacob Devlin", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"], "authorIds": ["39172707", "1744179", "2544107", "3259253"], "citationCount": 118946, "influentialCitationCount": 22739, "referenceCount": 63, "fieldsOfStudy": ["Computer Science"], "publicationTypes": ["JournalArticle", "Conference"], "isOpenAccess": true, "openAccessPdfUrl": "https://doi.org/10.18653/v1/n19-1423", "openAccessStatus": "GOLD", "openAccessLicense": "CCBY", "citations": null, "url": "https://www.semanticscholar.org/paper/df2b0e26d0599ce3e70df8a9da02e51594e0e992", "scrapedAt": "2026-08-06T12:41:07.882Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** With no API key, a lost query is the most likely reason a corpus comes back short. ```json { "requestsMade": 4, "requestsFailed": 1, "failures": [ { "request": "paper DOI:10.9999/not-a-real-doi", "error": "not found (404)" } ], "searchesRun": 3, "paperLookups": 1, "papersSaved": 421, "usedApiKey": true, "filters": { "searchQueries": ["retrieval augmented generation", "vector database embedding search"], "paperIds": ["DOI:10.9999/not-a-real-doi"], "fields": ["abstract", "tldr", "authors"], "includeCitations": false, "year": "2022-2026", "minCitationCount": 25, "openAccessOnly": true, "maxResults": 200 }, "finishedAt": "2026-08-06T12:44:51.300Z" } ``` `requestsMade` is searches plus lookups, one unit each regardless of how many pages a search walked. `failures[].request` is a label, `search ""` or `paper `, so it tells you exactly which input was lost. `papersSaved` is after deduplication and will be lower than the sum of the per-query caps whenever queries overlap. `filters.paperIds` shows the **normalised** identifiers, which is the place to confirm that a pasted URL resolved the way you intended. `usedApiKey: false` alongside failures is the usual explanation for a thin run. ## Behaviour to plan around - **Unticking a field looks exactly like missing data.** With `authors` left out of `fields`, every record has `authors: []` and `authorIds: []`; without `journal`, all three journal fields are `null`. Nothing marks the difference. Read `RUN_SUMMARY.filters.fields` before concluding that a paper has no authors. - **Deduplication is global.** A paper matching two searches, or matched by a search and also named in `paperIds`, is saved once. Per-query counts therefore do not add up to `papersSaved`, and the row carries no record of which query found it. - **A search reaches 999 papers and no further**, so `maxResults` is clamped there without an error. To go deeper, split the topic into narrower queries or slice it by `year` and run several. - **`year`, `minCitationCount` and `openAccessOnly` are search-only.** A paper named in `paperIds` is saved even if it violates all three. - **`citations` is capped upstream** at roughly the first thousand citing papers per record, and is not ordered by importance. For a heavily cited paper it is a sample, not the citation list, so never derive a count from its length; `citationCount` is the count. - **`influentialCitationCount` is a classifier output, not a subset you can inspect.** It answers "how often did this work actually shape the citing paper", and it is usually a small fraction of `citationCount`. - **`publicationDate` is often `null` where `year` is set.** Sort on `year` and treat `publicationDate` as a bonus, or a chronological ordering will silently drop conference papers. - **One failing request never aborts the run.** It is recorded in `RUN_SUMMARY.failures` and the rest continue. The Actor only throws when every request failed. - **Retries distinguish fatal from transient.** A malformed request and an unknown paper fail immediately, because retrying cannot help. Throttling and server errors back off on a ramp, honouring a retry-after hint where one is given. - **`doi` comes back bare, `paperIds` accepts it either way.** Round-tripping a dataset back into a later run needs no prefixing. ## Recipes **Systematic review sweep.** Several angles on one topic, bounded and filtered. ```json { "searchQueries": [ "retrieval augmented generation", "vector database embedding search", "mixture of experts language model" ], "year": "2022-2026", "minCitationCount": 25, "openAccessOnly": true, "maxResults": 200, "apiKey": "…" } ``` Rows are already deduplicated. Screen on `title` plus `abstract`, then fetch `openAccessPdfUrl` for anything that survives. **Enrich a list of identifiers you already hold.** Mixed forms are fine in one array. ```json { "paperIds": ["10.18653/v1/N19-1423", "arXiv:1706.03762", "PMID:19872477"], "fields": ["abstract", "tldr", "authors", "journal"] } ``` Join your own table to the result on `doi`, `arxivId` or `pubmedId`; all three come back bare, with no prefix. **Citation tracking on a fixed set.** Small `fields` list, run on a schedule. ```json { "paperIds": ["df2b0e26d0599ce3e70df8a9da02e51594e0e992"], "fields": [], "includeCitations": false, "apiKey": "…" } ``` Store `paperId`, `citationCount`, `influentialCitationCount` and `scrapedAt` per run and plot the difference. An empty `fields` array keeps the payload minimal. **Citation graph seed.** One expensive run, then work offline. ```json { "paperIds": ["df2b0e26d0599ce3e70df8a9da02e51594e0e992"], "includeCitations": true, "fields": ["authors", "fieldsOfStudy"], "apiKey": "…" } ``` Feed `citations[].paperId` back in as `paperIds` on a second run to expand one level out. Remember the citing list is capped, so this walks a sample of the graph.