--- name: world-bank-indicators-scraper description: Export World Bank development indicator series as flat records via the Apify Actor arman-bd/world-bank-indicators-scraper. Returns one row per country, indicator and year with the ISO code, country or aggregate name, indicator code and full name, the observation value, the publisher's unit and display precision, and the World Bank region and income classification joined on from the country reference table. Use when a task needs macro or country-risk time series, market sizing by population and GDP per capita, an economics dashboard refresh, or development and ESG series with region and income level already attached. Not for sub-national data, quarterly or monthly frequencies, forecasts, indicator discovery or search, or any series the World Bank does not publish. --- # World Bank Indicators Scraper Apify Actor `arman-bd/world-bank-indicators-scraper`. Give it indicator codes, optionally narrowed to countries and years, and get one dataset record per country-indicator-year. It runs without credentials. Rows come back up to 20,000 at a time, so a whole indicator across every country and year is usually one round trip. ## When to use it - Macro research and country-risk scoring: GDP, inflation and debt for a watchlist in one run. - Market sizing, joining population and GDP per capita to your own segmentation. - A scheduled refresh that keeps a time-series store current with no key management. - Development and ESG analysis, where `region` and `incomeLevel` arrive already attached to every row. - Reproducible data-journalism extracts, with `indicatorCode` recorded on every row so the pull can be repeated exactly. ## When not to use it - Sub-national data: cities, states, provinces. Country and aggregate level only. - Anything faster than annual. Every observation is a year. - Forecasts or projections. This is published history. - Finding an indicator. There is no search here; you must already know the code, or read it out of the indicator's page URL. - Series the World Bank does not carry. A code it does not recognise fails that indicator rather than returning an empty set. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/world-bank-indicators-scraper').call({ countries: ['USA', 'GBR', 'DEU'], indicators: ['NY.GDP.MKTP.CD', 'SP.POP.TOTL'], fromYear: 2020, skipNullValues: true, maxResults: 0, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); if (summary.countrySelector === 'all') console.warn('country filter did not survive'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~world-bank-indicators-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"countries":["USA","GBR"],"indicators":["NY.GDP.MKTP.CD"],"fromYear":2015}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/world-bank-indicators-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `indicators` | string[] | yes | | Indicator codes such as `NY.GDP.MKTP.CD`, or a full indicator page URL, from which the code is extracted. Upper-cased and deduplicated. An entry with no usable code is dropped; if none survive, the run throws. | | `countries` | string[] | no | `[]` | ISO-2 or ISO-3 codes. Aggregates such as `EMU` or `WLD` work. Empty, or `all` anywhere in the list, means every country and aggregate. | | `fromYear` | integer | no | none | Earliest year, minimum 1960. Setting only this fills the other end with the current year. | | `toYear` | integer | no | none | Latest year, minimum 1960. Setting only this fills the other end with 1960. Earlier than `fromYear` throws. | | `skipNullValues` | boolean | no | `true` | The source emits a row per country-year whether or not an observation exists. On drops them; off keeps a dense, gap-marked grid. | | `maxResults` | integer | no | `0` | Cap on total records across every indicator. `0` means no cap. | **Row volume, not request count, is what you are managing.** One indicator across every country and every year is roughly 17,000 rows and typically one request, so adding indicators costs about one request each and adding countries or years costs nothing extra. That makes the natural shape of a run "many indicators, no cap". The one setting that will surprise you is `maxResults`: it is a global cap that stops the entire run the moment it is hit, mid-indicator, so the indicators later in your list get no rows at all and nothing is recorded as a failure. Either leave it at `0` or set it high enough to cover every indicator you asked for. `skipNullValues: false` roughly doubles or triples row counts on sparse series, so pair it with a country and year filter rather than with `all`. ## Output One record per country, indicator and year. | Field | Type | Notes | |---|---|---| | `countryIso` | string \| null | ISO-3 where the source has one, falling back to ISO-2 on the handful of aggregate rows that lack it. Join key. | | `countryName` | string \| null | Country or aggregate name as published, with trailing whitespace trimmed. | | `indicatorCode` | string | The code the source echoed back, falling back to the code you asked for. | | `indicatorName` | string \| null | Full series name, for example `GDP (current US$)`. The unit is usually embedded here. | | `year` | number \| null | Observation year, as a number, not a string. | | `value` | number \| null | The observation. `null` where no data exists, which only appears when `skipNullValues` is off. | | `unit` | string \| null | Almost always `null`; the source ships the unit inside `indicatorName` instead. Preserved for the rare series that populate it. | | `region` | string \| null | World Bank region, joined on from the country reference table. `Aggregates` on non-country rows. `null` if that lookup failed. | | `incomeLevel` | string \| null | Income classification such as `Lower middle income`. `Aggregates` on non-country rows. Same failure mode as `region`. | | `decimalPlaces` | number \| null | The publisher's suggested display precision. Not a statement about the value's accuracy. | | `scrapedAt` | string | Run timestamp, ISO 8601. | A real record: ```json { "countryIso": "USA", "countryName": "United States", "indicatorCode": "NY.GDP.MKTP.CD", "indicatorName": "GDP (current US$)", "year": 2025, "value": 30769700000000, "unit": null, "region": "North America", "incomeLevel": "High income", "decimalPlaces": 0, "scrapedAt": "2026-08-06T12:00:00.000Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "indicatorsRequested": ["NY.GDP.MKTP.CD", "SP.POP.TOTL"], "indicatorsFailed": 0, "failures": [], "countrySelector": "USA;GBR;DEU", "recordsSaved": 36, "filters": { "countries": ["USA", "GBR", "DEU"], "fromYear": 2020, "toYear": null, "skipNullValues": true, "maxResults": 0 }, "finishedAt": "2026-08-06T12:00:04.412Z" } ``` **`countrySelector` is the field to check first.** It is what was actually asked for, and if it reads `all` when you passed a country list, every one of your codes was rejected during validation and you have just pulled the whole world. `indicatorsRequested` is the list of codes that survived normalisation, so compare its length against your input. `indicatorsFailed` counts distinct indicators with at least one failure, while `failures` has one entry per failed page carrying `indicator`, `page` and `error`, so one indicator can appear several times. `filters.fromYear` and `filters.toYear` show `null` for an open end even though the query sent a filled-in bound. ## Behaviour to plan around - **A country name silently becomes "every country".** Validation accepts only two or three letter codes, so `["United States", "Germany"]` is rejected entry by entry, the list ends up empty, and an empty list means `all`. You get 200-plus countries and every aggregate instead of two. Check `RUN_SUMMARY.countrySelector`. - **`all` beats a specific list.** The source treats them as mutually exclusive, so `all` anywhere in `countries` discards the rest. - **Aggregates are mixed in with countries** when you ask for `all`: rows such as `WLD` (World), `EMU` (euro area) and `AFE` (Africa Eastern and Southern) are not sovereign states. They are marked `region: "Aggregates"` and `incomeLevel: "Aggregates"`, so filter on `region !== "Aggregates"` to get countries only. Summing over an unfiltered `all` pull double-counts badly. - **A bad code arrives as a success at the transport level.** An unrecognised indicator or country is reported in the response body rather than as an error status, and the Actor turns that into a failure for that indicator with the source's own message. It is treated as final and not retried. - **`region` and `incomeLevel` are joined, not returned.** They come from a reference table fetched once per run. If that single lookup fails the run continues and both fields are `null` on every row, with a warning in the log and nothing in `failures`. - **`maxResults` stops the whole run, not each indicator.** Later indicators in your list silently contribute nothing. - **`unit` being `null` is normal**, not a gap. Read the unit out of `indicatorName`. - **The most recent year is missing for many countries.** Publication lag varies by country and series. That is the source's cadence, not a truncated pull. - **One failing indicator never aborts the run.** The Actor throws only when every indicator failed and nothing was saved. Transient conditions get four attempts with exponential backoff. - **Whitespace is trimmed on joined reference values**, so `region` values are safe to group on directly. ## Recipes **Watchlist macro pull.** Several indicators, a handful of countries, no cap. ```json { "countries": ["USA", "GBR", "DEU", "JPN", "CHN"], "indicators": ["NY.GDP.MKTP.CD", "FP.CPI.TOTL.ZG", "SL.UEM.TOTL.ZS"], "fromYear": 2000, "maxResults": 0 } ``` Pivot on `countryIso` and `year`, one column per `indicatorCode`. **One indicator, everyone, this century.** About 5,000 rows after nulls are dropped. ```json { "countries": ["all"], "indicators": ["FP.CPI.TOTL.ZG"], "fromYear": 2000, "toYear": 2024 } ``` Drop `region === "Aggregates"` before ranking countries, or the World row wins. **Dense grid for time-series work.** Keeps the missing years present so gaps are explicit rather than implied. ```json { "countries": ["BD"], "indicators": ["NY.GDP.PCAP.CD"], "skipNullValues": false } ``` Every year from 1960 appears, with `value: null` where nothing was published. **Rule of thumb for sizing.** Estimate roughly 65 years times the number of countries times the number of indicators, then halve it if `skipNullValues` is on. `all` is 295 entities, so one indicator over the full history is around 17,000 rows before nulls are dropped. If that fits, leave `maxResults` at `0`.