--- name: wikipedia-articles-scraper description: Fetch Wikipedia articles as structured records via the Apify Actor arman-bd/wikipedia-articles-scraper. Returns the resolved title, page id, language, one-line short description, clean plain-text lead summary, lead image at thumbnail and original size, coordinates, last-modified timestamp and canonical URL, plus optional full article text, outbound links and categories, from any language edition. Use when a task needs a licence-clean corpus for retrieval, entity enrichment with a canonical description and image, multilingual coverage comparison, geodata for many places, or link-graph analysis. Not for infobox fields as structured data, revision history, edit metadata, article search or discovery. --- # Wikipedia Articles Scraper Apify Actor `arman-bd/wikipedia-articles-scraper`. Give it article titles, slugs or Wikipedia URLs, get one dataset record per article per language. Text comes back as real plain text, not wikitext and not stripped markup. No credentials are involved. ## When to use it - Building a retrieval corpus: `includeFullText`, then chunk and embed. - Entity enrichment: attaching a canonical one-line `description` and a lead image to records you already have. - Multilingual comparison: the same title across `en`, `de`, `fr` and `ja`, diffed on `extract` length or on presence at all. - Geodata: `coordinates` for thousands of places in one run. - Link-graph analysis over a topic cluster, via `includeLinks`. ## When not to use it - Infobox fields as structured data. The infobox is rendered wikitext, not a data structure. `description`, `coordinates`, `thumbnail` and `originalImage` cover the high-value facts; for the rest, use a Wikidata Actor. - Revision history, diffs, editors, talk pages or pageview statistics. - Search or discovery. You must already know the titles; nothing here finds them. - Wiktionary, Wikisource, Commons, Wikivoyage or any non-Wikipedia project. - Non-article namespaces. Category, Template and Talk pages are not the target. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/wikipedia-articles-scraper').call({ titles: ['Web scraping', 'Berlin', 'https://de.wikipedia.org/wiki/Kartoffel'], languages: ['en'], includeFullText: true, includeLinks: false, includeCategories: true, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~wikipedia-articles-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"titles":["Web scraping","Berlin"],"languages":["en"],"includeFullText":true}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/wikipedia-articles-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `titles` | string[] | yes | | Titles (`Web scraping`), slugs (`web_scraping`, underscores become spaces) or full Wikipedia article URLs, including mobile and `index.php?title=` forms. Mixed input is fine. A URL names its own wiki and is fetched only from that one. An empty or wholly unparseable list throws. | | `languages` | string[] | no | `["en"]` | Language codes, lowercased. Every *plain* title is fetched once per language, so two titles across two languages is four records. An empty array falls back to `["en"]`. Any edition code works, including `simple`. | | `includeFullText` | boolean | no | `false` | Add `fullText`, the complete article as plain text. | | `includeLinks` | boolean | no | `false` | Add `links`, the outbound main-namespace article titles, first 500. | | `includeCategories` | boolean | no | `false` | Add `categories`, visible memberships with the namespace prefix stripped. Hidden maintenance categories are excluded. | **`includeFullText` is the one switch that changes the shape of a run.** The summary for each article is always one request. `includeLinks` and `includeCategories` are gathered 50 articles at a time, so they add almost nothing to a large run. Full text cannot be batched at all, so switching it on adds one request per article and turns a 500-article run from about eleven requests of extras into 500. Ask for full text on the lists where you genuinely need the body, and take `description` and `extract` alone when you are enriching at scale. ## Output One record per resolved article per language. The three optional blocks are omitted from the record entirely when their switch is off. | Field | Type | Notes | |---|---|---| | `title` | string | The *resolved*, normalised title. Redirects are followed, so `NYC` comes back as `New York City`. This is not necessarily what you passed in. | | `pageId` | number \| null | Page id within this language edition. Ids are not comparable across editions. | | `lang` | string | Language code this record came from. | | `description` | string \| null | One-line short description, the grey line under the title. `null` on articles that have none. | | `extract` | string \| null | Clean plain-text lead section. Falls back to a decoded and stripped HTML variant on wikis that only publish that. | | `fullText` | string \| null | Complete article as plain text. Present only when `includeFullText` is on, and `null` within such a run if the extras request failed for that article. | | `thumbnail` | object \| null | Lead image, `{ source, width, height }`. `null` when the article has no image. | | `originalImage` | object \| null | Same image at full size, `{ source, width, height }`. | | `coordinates` | object \| null | `{ lat, lon }` for geotagged articles. Filled from the extras request when the summary did not carry it. | | `categories` | string[] | Present only when `includeCategories` is on. Prefix stripped, hidden categories excluded. Empty array when the extras request failed. | | `links` | string[] | Present only when `includeLinks` is on. Outbound article titles, main namespace, capped at 500 with no paging beyond that. | | `lastModified` | string \| null | Timestamp of the revision you received, ISO 8601. Diff on this to detect edits. | | `url` | string | Canonical desktop article URL. | | `scrapedAt` | string | Run timestamp, ISO 8601 UTC. | A real record, long strings trimmed: ```json { "title": "Berlin", "pageId": 3354, "lang": "en", "description": "Capital and largest city of Germany", "extract": "Berlin is the capital and largest city of Germany. With 3.7 million inhabitants, …", "fullText": "Berlin is the capital and largest city of Germany…\n\nHistory\n\nEtymology\n\n…", "thumbnail": { "source": "https://upload.wikimedia.org/wikipedia/commons/thumb/f/f7/Museumsinsel_Berlin_Juli_2021_1_%28cropped%29_b.jpg/330px-…jpg", "width": 330, "height": 192 }, "originalImage": { "source": "https://upload.wikimedia.org/wikipedia/commons/f/f7/Museumsinsel_Berlin_Juli_2021_1_%28cropped%29_b.jpg", "width": 5464, "height": 3186 }, "coordinates": { "lat": 52.52, "lon": 13.405 }, "categories": ["Berlin", "Capitals in Europe", "German state capitals"], "links": [".berlin", "1. FC Union Berlin", "Aachen"], "lastModified": "2026-08-01T21:11:05Z", "url": "https://en.wikipedia.org/wiki/Berlin", "scrapedAt": "2026-08-06T11:42:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "articlesRequested": 6, "articlesSaved": 5, "articlesFailed": 1, "failures": [ { "source": "de:Web scraping", "error": "article not found (404)" } ], "filters": { "languages": ["en", "de"], "includeFullText": true, "includeLinks": false, "includeCategories": true }, "finishedAt": "2026-08-06T11:42:02.318Z" } ``` `articlesRequested` is the number of language-and-title pairs after de-duplication, not the length of your `titles` array. **`articlesFailed` is not simply the shortfall in `articlesSaved`:** an extras request that fails is counted there too, even though the article was saved with its summary intact. Tell the two apart by the `error` prefix, since an extras failure reads `extras: …`. Each `source` is formatted `lang:title` using the title as you asked for it. `filters` does not echo `titles`, so keep your own list. ## Behaviour to plan around - **The output has no field for what you asked for.** Redirects and normalisation mean `title` is the resolved article, so `NYC` returns `New York City` and there is no record of the input string. Keep your own mapping, and match on `title` plus `lang` only after resolving. - **A URL in `titles` ignores `languages` entirely.** It carries its own edition, so a run of three plain titles across four languages plus one German URL produces 13 records, not 16. - **Record count is titles times languages.** The multiplication is the main way a run gets unexpectedly large, and full text multiplies request count on top of it. - **A missing article in one language is not a failed run.** Coverage differs between editions, so `de:Some topic` failing while `en:Some topic` succeeds is normal and informative; that is the whole point of a coverage diff. - **Extras fail as a batch, not per article.** When a batched extras request fails, every article in that chunk gets a failure entry and keeps `fullText` `null` with `categories` and `links` empty. Empty arrays are therefore ambiguous: check `failures` before concluding an article has no categories. - **`links` stops at 500** and there is no paging past it, so a heavily linked article is truncated silently. Treat it as a sample of the link graph, not the whole of it. - **`categories` excludes hidden maintenance categories** and has the namespace prefix removed, so it will not match raw category names you have from elsewhere. - **`pageId` is edition-scoped.** The same subject has different ids in `en` and `de`. Never join across languages on it. - **Extras are matched by page id, not by title**, so a redirect can never attach one article's text to another's record. - **Transient failures are retried** four times with exponential backoff and jitter. A missing article is permanent and is not retried. - **The run throws only when nothing at all was saved.** ## Recipes **Corpus build for retrieval.** Full text plus categories for filtering, no links. ```json { "titles": ["Web scraping", "Data mining", "Information extraction"], "languages": ["en"], "includeFullText": true, "includeCategories": true, "includeLinks": false } ``` Chunk `fullText` on the blank-line boundaries, which mark section breaks, and keep `url` and `lastModified` on each chunk for citation and staleness checks. **Entity enrichment at scale.** Nothing optional switched on, so it is one request per article and the rows stay small. ```json { "titles": ["Berlin", "Lyon", "Osaka", "Recife"], "languages": ["en"] } ``` Take `description` for a label, `thumbnail.source` for an image and `coordinates` for a map pin. `extract` is there when you want a longer blurb. **Multilingual coverage diff.** One topic, several editions. ```json { "titles": ["Artificial intelligence"], "languages": ["en", "de", "fr", "ja", "simple"] } ``` Compare `extract` length per `lang`, and read `RUN_SUMMARY.failures` for the editions where the article does not exist at all. That absence is the finding. **Link-graph seed crawl.** Links on, full text off, so the extras stay batched. ```json { "titles": ["Graph theory"], "languages": ["en"], "includeLinks": true } ``` Feed the returned `links` back in as `titles` for a second run to walk one hop out. Remember the 500-link ceiling before treating the result as a complete neighbourhood.