--- name: arxiv-papers-scraper description: Search arXiv in its native query syntax via the Apify Actor arman-bd/arxiv-papers-scraper and get one structured record per preprint. Returns title, abstract, author list, primary and cross-listed categories, DOI, journal reference, submission and revision dates, and direct PDF and abstract links. Use when a task needs a literature sweep, a daily feed of new preprints in a subject area, an author or research-group watchlist, or abstracts to embed into a RAG index. Not for full paper text, citation counts, journals outside arXiv, or author disambiguation. --- # arXiv Scraper: Preprints, Authors & Categories Apify Actor `arman-bd/arxiv-papers-scraper`. Give it arXiv search queries, categories, or both, and get one dataset record per paper: metadata plus the full abstract. It takes no credentials. arXiv's own query syntax is passed through untouched, so anything that works in arXiv's advanced search works here. ## When to use it - A literature sweep on a topic, where you want abstracts rather than titles alone. - A scheduled "what landed overnight" feed for one or more subject categories. - Watching a research group: one query per author, results merged and de-duplicated. - Building a RAG or embedding corpus, where `abstract`, `categories` and `absUrl` are the whole payload you need. - Checking whether a batch of preprints has since been formally published, via `doi` and `journalRef`. ## When not to use it - Full paper text. You get the abstract and a link to the PDF, nothing more. - Citation counts, references, h-index or any bibliometric graph. arXiv does not carry them and the Actor invents nothing. - Literature that is not on arXiv. There is no fallback to another index. - Author identity resolution. `authors` is a list of name strings exactly as submitted, with no ORCID and no disambiguation between two people of the same name. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/arxiv-papers-scraper').call({ searchQueries: ['all:"large language model" AND ti:agent'], categories: ['cs.LG', 'cs.CL'], fromDate: '2026-01-01', sortBy: 'submittedDate', maxResultsPerQuery: 25, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~arxiv-papers-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchQueries":["cat:cs.LG AND all:transformer"],"sortBy":"submittedDate","maxResultsPerQuery":25}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/arxiv-papers-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchQueries` | string[] | conditional | `[]` | arXiv native syntax. Prefixes `all:`, `ti:`, `abs:`, `au:`, `cat:`, `jr:`, `co:`. Combine with `AND` / `OR` / `ANDNOT`, group with parentheses, quote multi-word phrases. Each entry is run and paginated separately. Duplicates are removed first. | | `categories` | string[] | conditional | `[]` | arXiv category codes such as `cs.LG`, `cs.CL`, `stat.ML`, `q-bio.NC`. ORed together, then ANDed onto every query. With no query, the categories alone become the search. | | `fromDate` | string | no | `""` | Submission-date floor, `YYYY-MM-DD` only. Anything else throws before the first request. There is no upper bound and no `toDate`. | | `sortBy` | string | no | `"submittedDate"` | One of `submittedDate`, `lastUpdatedDate`, `relevance`. Always descending. An unrecognised value throws. | | `maxResultsPerQuery` | integer | no | `100` | Papers saved per query, not per run. `0` means no limit. Minimum `0`. | **At least one of `searchQueries` or `categories` must be non-empty**, or the run throws before any request goes out. The real decision is breadth against wall-clock time: requests are paced, so a run's duration is roughly three seconds per 200 papers per query on top of transfer. A category-only search is the cheapest way to pull a whole subject feed, and `fromDate` with `sortBy: submittedDate` is the combination that keeps a scheduled run bounded no matter how large the category is. Raising `maxResultsPerQuery` past about 30,000 is the wrong lever: deep offsets degrade, so slice the same ground into date windows with `fromDate` instead. ## Output One record per paper, de-duplicated across every query in the run. | Field | Type | Notes | |---|---|---| | `arxivId` | string | Identifier **including the version suffix**, `2608.04828v1`, or an old-style `cond-mat/0102536v1`. | | `title` | string \| null | Line wrapping removed, whitespace collapsed. | | `abstract` | string \| null | Full abstract as one paragraph, XML entities decoded. | | `authors` | string[] | Names in submission order. Plain strings, no affiliations. | | `primaryCategory` | string \| null | The single category the authors filed under. | | `categories` | string[] | Every category it is cross-listed in, primary included, de-duplicated. | | `published` | string \| null | ISO 8601 timestamp of the v1 submission. | | `updated` | string \| null | ISO 8601 timestamp of the latest revision. Equals `published` for a paper never revised. | | `doi` | string \| null | Publisher DOI once the paper is formally published. `null` while preprint-only. | | `journalRef` | string \| null | Free-text journal citation, e.g. `J. Chem. Phys. 115, 1626 (2001)`. | | `comment` | string \| null | Author's note: page count, figures, conference acceptance. | | `pdfUrl` | string \| null | Direct PDF link. Falls back to a link built from `arxivId`. | | `absUrl` | string \| null | Abstract landing page. Same fallback. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record, long strings trimmed: ```json { "arxivId": "2608.04828v1", "title": "Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?", "abstract": "Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. …", "authors": ["Jinyi Han", "Yuanjian Xu", "Ying Liao", "Xinyi Wang", "Zishang Jiang"], "primaryCategory": "cs.CL", "categories": ["cs.CL", "cs.AI"], "published": "2026-08-05T13:29:16Z", "updated": "2026-08-05T13:29:16Z", "doi": null, "journalRef": null, "comment": "18 pages, 6 figures", "pdfUrl": "https://arxiv.org/pdf/2608.04828v1", "absUrl": "https://arxiv.org/abs/2608.04828v1", "scrapedAt": "2026-08-06T11:28:08.231Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "queriesRequested": 2, "queriesFailed": 1, "failures": [ { "query": "all:not_a_real_term_zzzq AND badsyntax:(", "error": "arXiv rejected the query (400), check the search syntax" } ], "papersSaved": 13, "filters": { "searchQueries": ["cat:cond-mat.str-el AND ti:cusp", "all:not_a_real_term_zzzq AND badsyntax:("], "categories": [], "fromDate": null, "sortBy": "relevance", "maxResultsPerQuery": 210 }, "finishedAt": "2026-08-06T11:31:44.902Z" } ``` `queriesFailed` above zero means the dataset is missing whatever those queries would have matched, and each entry in `failures` names the query and why. Bad syntax is the usual cause and re-running it unchanged will fail again. `papersSaved` is the count after global de-duplication, so it is legitimately lower than `queriesRequested * maxResultsPerQuery` even on a fully successful run. `filters` echoes the input as the Actor understood it, which is the fastest way to catch a `fromDate` that silently stayed empty or a category list that never made it through. ## Behaviour to plan around - **`arxivId` carries the version suffix.** `2608.04828v1` and `2608.04828v2` are different strings for the same paper. Diffing raw `arxivId` between two scheduled runs reports every revision as a new paper. Strip the trailing `v\d+` before you key on it, and keep the full value if you care about which revision you read. - **`maxResultsPerQuery` is per query.** Five queries at 100 is a budget of 500 fetched papers, not 100. - **De-duplication is global.** A paper matched by several queries is saved once, so the dataset row count is usually below the sum of the per-query caps. - **A failing query never aborts the run.** It is recorded in `RUN_SUMMARY.failures` and the remaining queries continue. The Actor only throws when every query failed, or when the input was invalid before the first request. - **Bad syntax is not retried.** A rejected query and a malformed feed fail fast, because another attempt cannot change the answer. Transient failures get three attempts with linear backoff. - **`doi`, `journalRef` and `comment` are null far more often than not.** arXiv only reports what the submitter supplied. A `null` DOI means "not recorded here", never "not published anywhere". - **`sortBy: lastUpdatedDate` surfaces old papers.** A 2019 preprint revised yesterday sorts above a paper submitted this morning. Use `submittedDate` for a genuine new-work feed. - **`fromDate` is a floor with no ceiling.** The range runs to the far future, so it cannot be used to isolate a closed window in one call. Run one call per window and join the results yourself. - **Requests are paced.** Throughput is bounded by design, so treat a large run as minutes rather than seconds and prefer scheduled narrow runs over one enormous one. ## Recipes **Daily new-preprint feed.** Category-only search with a one-day floor, scheduled every morning. ```json { "categories": ["cs.LG", "cs.CL", "cs.AI"], "fromDate": "2026-08-05", "sortBy": "submittedDate", "maxResultsPerQuery": 500 } ``` De-duplicate against yesterday on `arxivId` with the version suffix stripped. **Build a RAG corpus for a topic.** Several phrasings of the same idea, fenced into the relevant categories, merged into one de-duplicated set. ```json { "searchQueries": [ "all:\"retrieval augmented generation\"", "all:\"vector database\" AND abs:embedding", "ti:\"mixture of experts\"" ], "categories": ["cs.CL", "cs.IR"], "maxResultsPerQuery": 300 } ``` Embed `title` plus `abstract`, store `absUrl` and `arxivId` as the citation key. **Track a research group.** One query per author. Co-authored papers appear once. ```json { "searchQueries": ["au:\"Yoshua Bengio\"", "au:\"Yann LeCun\"", "au:\"Geoffrey Hinton\""], "sortBy": "submittedDate", "maxResultsPerQuery": 50 } ``` Group by `primaryCategory` to see where the group's output is landing. **Check preprints for publication.** Re-run a known set and look at what filled in. ```json { "searchQueries": ["cat:cond-mat.str-el AND ti:cusp"], "sortBy": "lastUpdatedDate", "maxResultsPerQuery": 100 } ``` Rows where `doi` or `journalRef` is now non-null have been published since you last looked.