--- name: podcast-directory-scraper description: Find podcasts and read their feeds as structured records via the Apify Actor arman-bd/podcast-directory-scraper. Returns one record per show with publisher, genres, artwork, episode count and feed URL, and optionally one record per episode with title, show notes as plain text, publish date, duration in seconds, season and episode numbers and the direct audio URL. Use when a task needs podcast discovery by keyword or Apple ID, a catalogue of episodes for a transcription or monitoring pipeline, or genre and cadence research. Not for audio files themselves, transcripts, listener numbers, chart rankings or private feeds. --- # Podcast Directory Scraper Apify Actor `arman-bd/podcast-directory-scraper`. Two steps chained: shows are discovered in Apple's public podcast directory, then each show's own RSS feed is read for episodes. It takes no credentials of your own. Records come in two shapes, distinguished by `recordType`. ## When to use it - Finding every show in a niche, with publisher names and feed URLs, for PR or guest sourcing. - Building a searchable index of shows by genre and episodes by title and show notes. - Feeding a transcription pipeline: `audioUrl` is a direct link to the audio file. - Monitoring a fixed set of shows on a schedule and picking up new episodes. - Research on episode cadence, duration trends or genre mix across a category. ## When not to use it - Downloading or transcoding the audio. You get the URL, the byte length and the MIME type; fetching the file is your step. - Transcripts. Podcast RSS has no structured transcript field, so unless a publisher pastes one into the show notes there is nothing to return. - Listener counts, downloads, revenue or chart positions. None of that is public. - Private, paywalled or subscriber-only feeds. Those fail and are recorded as failures. - Exhaustive discovery of a broad topic. A single search returns at most 100 shows and there is no pagination. ## Call it ```js import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('arman-bd/podcast-directory-scraper').call({ podcastIds: ['1200361736', 'https://podcasts.apple.com/us/podcast/serial/id917918570'], includeEpisodes: true, maxEpisodesPerShow: 25, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); const { value: summary } = await client .keyValueStore(run.defaultKeyValueStoreId) .getRecord('RUN_SUMMARY'); ``` One-shot over HTTP, when you want the rows back in the same request: ```bash curl -X POST "https://api.apify.com/v2/acts/arman-bd~podcast-directory-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"searchTerms":["history"],"maxShowsPerTerm":25,"includeEpisodes":true,"maxEpisodesPerShow":5}' ``` The Actor is also exposed through Apify's MCP server as `arman-bd/podcast-directory-scraper`, so an MCP-capable agent can call it with no extra wiring. ## Input | Field | Type | Required | Default | Notes | |---|---|---|---|---| | `searchTerms` | string[] | no | `[]` | Keywords for the directory: a topic, a show name or a publisher. One request each, at most 100 shows per term. Deduplicated. | | `podcastIds` | string[] | no | `[]` | Apple collection IDs, or Apple Podcasts URLs the ID is extracted from. Looked up 50 per request. Anything without a recognisable ID is dropped. | | `feedUrls` | string[] | no | `[]` | RSS feed URLs read directly, with no directory lookup. Show metadata then comes from the feed's own channel tags. A missing scheme is filled in. | | `country` | string | no | `"US"` | Two-letter storefront code, upper-cased. Results and ranking differ by market. Anything not two letters throws. No effect on `feedUrls`. | | `maxShowsPerTerm` | integer | no | `20` | Shows taken from each search, 1 to 100. Values above 100 are clamped, because 100 is the ceiling a single search can return. | | `includeEpisodes` | boolean | no | `false` | Fetch each show's feed and emit one record per episode. One extra request per show. | | `maxEpisodesPerShow` | integer | no | `10` | Episodes saved per show, newest first, 0 to 5000. `0` means the whole feed. | **`includeEpisodes` is what turns a cheap run into a long one.** Discovery alone is one request per search term and one per 50 IDs, so a 100-show sweep is a couple of requests and finishes in seconds. Turning episodes on adds one feed fetch per show, paced apart, so the same sweep becomes a hundred requests against a hundred different publishers. Keep it off while you are still deciding which shows you want, then turn it on for the shortlist with a small `maxEpisodesPerShow`, since feeds are newest-first and long-running shows carry thousands of items. ## Output Every record carries the show fields. Episode records fill in the episode fields as well; on show records those are `null`. | Field | Type | Notes | |---|---|---| | `recordType` | string | `show` or `episode`. Branch on this first. | | `source` | string | What produced the record: `search:`, `id:` or `feed:`. | | `collectionId` | number \| null | Apple's ID for the show. `null` for direct feeds. | | `showName` | string \| null | Show title. | | `artistName` | string \| null | Publisher. | | `feedUrl` | string \| null | The show's RSS feed. `null` when the directory lists no feed. | | `feedTitle` | string \| null | Title inside the feed itself. `null` unless the feed was actually fetched. | | `genres` | string[] | Directory genre list. Empty array for direct feeds. | | `primaryGenre` | string \| null | Primary genre, or the feed's first category for direct feeds. | | `episodeCount` | number \| null | The directory's episode count. `null` for direct feeds. Not the number of rows you got. | | `artworkUrl` | string \| null | Show artwork, 600 px where available. | | `country` | string \| null | The directory's country for the show, a three-letter code such as `USA`. Falls back to the two-letter storefront you passed. Do not assume one format. | | `contentAdvisoryRating` | string \| null | For example `Clean` or `Explicit`. | | `appleUrl` | string \| null | The show's directory page. `null` for direct feeds. | | `episodeGuid` | string \| null | The feed's own identity for the episode. Episode records only. | | `title` | string \| null | Episode title on episode records, show name on show records. | | `link` | string \| null | Episode link, falling back to the audio URL. On show records, the directory page or the feed URL. | | `author` | string \| null | Episode byline, falling back to the publisher. | | `publishedAt` | string \| null | ISO 8601. Episode publish time; on show records, the show's most recent release date. | | `contentText` | string \| null | Show notes as clean plain text on episodes, the channel description on shows. `null` on a show record whose feed was not fetched. | | `duration` | string \| null | The feed's duration string, verbatim. | | `durationSeconds` | number \| null | Parsed from `HH:MM:SS`, `MM:SS` or a bare second count. `null` when the feed omits it or the value is unparseable. | | `audioUrl` | string \| null | Direct link to the audio file, from the episode's enclosure. | | `audioLengthBytes` | number \| null | Byte size declared in the enclosure. Publishers sometimes declare it wrongly. | | `audioType` | string \| null | MIME type, usually `audio/mpeg`. | | `episodeNumber` | number \| null | Only when the publisher tags it. | | `seasonNumber` | number \| null | Only when the publisher tags it. | | `episodeType` | string \| null | `full`, `trailer` or `bonus`. | | `explicit` | string \| null | The raw feed value, so a **string** such as `"true"`, `"false"`, `"yes"` or `"clean"`. Not a boolean. | | `imageUrl` | string \| null | Episode artwork, falling back to the show artwork. | | `scrapedAt` | string | Run timestamp, ISO 8601. | An episode record, long strings trimmed: ```json { "recordType": "episode", "source": "id:917918570", "collectionId": 917918570, "showName": "Serial", "artistName": "Serial Productions & The New York Times", "feedUrl": "https://feeds.simplecast.com/PpzWFGhg", "feedTitle": "Serial", "genres": ["News", "Podcasts", "True Crime"], "primaryGenre": "News", "episodeCount": 125, "artworkUrl": "https://is1-ssl.mzstatic.com/image/thumb/Podcasts221/v4/…/600x600bb.jpg", "country": "USA", "contentAdvisoryRating": "Clean", "appleUrl": "https://podcasts.apple.com/us/podcast/serial/id917918570?uo=4", "episodeGuid": "20ad2ce9-3088-449b-8001-2e3e739b54d8", "title": "The Last 12 Weeks - Ep. 5", "link": "https://serialpodcast.org", "author": "Serial Productions & The New York Times", "publishedAt": "2026-06-18T10:20:00.000Z", "contentText": "Days before the execution, the defense team scrambles to respond to an unexpected ruling …", "duration": "00:39:22", "durationSeconds": 2362, "audioUrl": "https://dts.podtrac.com/redirect.mp3/…/audio/128/default.mp3", "audioLengthBytes": 37796581, "audioType": "audio/mpeg", "episodeNumber": 5, "seasonNumber": 17, "episodeType": "full", "explicit": "true", "imageUrl": "https://image.simplecastcdn.com/images/…/3000x3000/album_art.jpg", "scrapedAt": "2026-08-06T11:52:18.721Z" } ``` ## RUN_SUMMARY Written to the run's key-value store under the key `RUN_SUMMARY`. **Read it.** It is where a partial run admits that it was partial. ```json { "searchTerms": ["true crime"], "podcastIds": ["1200361736", "917918570"], "directFeeds": [], "showsSaved": 51, "episodesSaved": 510, "requestsFailed": 1, "failures": [ { "source": "id:917918570", "feedUrl": "https://feeds.example.com/x", "error": "feed is private (HTTP 403)" } ], "filters": { "country": "US", "maxShowsPerTerm": 50, "includeEpisodes": true, "maxEpisodesPerShow": 10 }, "finishedAt": "2026-08-06T11:53:02.117Z" } ``` `searchTerms`, `podcastIds` and `directFeeds` echo what was actually used after normalisation, which is the first thing to check when the result looks unexpectedly small: an ID with no digits, or a term that collapsed to nothing, never reaches the directory. `showsSaved` below the number of IDs you passed usually means those IDs do not exist, and that case is reported in the run log rather than in `failures`. Feed problems do land in `failures`, with a `feedUrl` alongside the `source`; a show whose feed failed is still saved as a show record with no episodes. `filters.maxEpisodesPerShow` is `null` when you asked for the whole feed. ## Behaviour to plan around - **An empty input is not an error.** With no terms, IDs or feed URLs, the Actor searches for `technology` so the run still returns something. If you get unexpected shows, check that your discovery input actually survived normalisation. - **A single search returns at most 100 shows and cannot be paged.** Asking for more is clamped. To cover a broad topic, split it into several narrower terms or supply IDs and feed URLs directly. - **Unknown IDs are silent.** An ID with no matching show simply produces no record and is noted in the run log, not in `failures`. Reconcile `collectionId` in the output against the IDs you sent. - **Shows are deduplicated by feed URL,** so the same show found by two different inputs is read once. A show the directory lists with no feed URL is not deduplicated and can appear twice, and it can never produce episodes. - **A direct feed URL is always fetched,** even with `includeEpisodes` off, because the channel is the only source of its metadata. A show found through the directory is only fetched when episodes are on, which is why `feedTitle` and `contentText` are `null` on discovery-only show records. - **`explicit` is a string, not a boolean,** because feeds use several spellings. Compare case-insensitively against `true` and `yes` rather than testing truthiness. - **`episodeCount` comes from the directory, not from the feed,** and will not match the number of episode rows you received once `maxEpisodesPerShow` is applied. It also drifts from reality for shows that prune old episodes. - **Episodes are newest first,** so a small `maxEpisodesPerShow` gives you the recent catalogue rather than an arbitrary slice. - **Feeds are inconsistent and missing values come back as `null`,** never as a guess. Absent durations, missing season numbers, malformed dates and enclosure-less items are all handled defensively. - **One failure never kills the run.** A failed search, a dead feed or a private feed is recorded and the run continues. The Actor only throws when nothing at all was saved. - **Directory calls and feed fetches are paced differently,** the former several seconds apart and the latter twice a second, so run time is dominated by the number of shows once episodes are on. ## Recipes **Discovery sweep, no episodes.** Fast and cheap, one record per show. ```json { "searchTerms": ["true crime", "software engineering"], "maxShowsPerTerm": 50 } ``` Filter on `primaryGenre` and `episodeCount`, then take the `collectionId` values worth following up. **Recent catalogue for a shortlist.** IDs plus episodes, capped. ```json { "podcastIds": ["1200361736", "https://podcasts.apple.com/us/podcast/serial/id917918570"], "includeEpisodes": true, "maxEpisodesPerShow": 25 } ``` Take rows where `recordType === "episode"` and hand `audioUrl` to a transcription step; `durationSeconds` tells you the cost before you start. **A feed you already know.** Skips the directory entirely, and the whole feed. ```json { "feedUrls": ["https://feeds.simplecast.com/Sl5CSM3S"], "includeEpisodes": true, "maxEpisodesPerShow": 0 } ``` Useful for shows that are not in the directory. Expect `collectionId`, `appleUrl` and `episodeCount` to be `null` on every row. **Daily monitoring.** A fixed list, only the newest few episodes each time. ```json { "podcastIds": ["1200361736", "917918570"], "includeEpisodes": true, "maxEpisodesPerShow": 5 } ``` Diff on `episodeGuid` against what you already hold; anything new is a fresh episode. **Finding an Apple ID.** It is the number after `id` in the show's Apple Podcasts URL, so `podcasts.apple.com/us/podcast/the-daily/id1200361736` gives `1200361736`. You can paste the whole URL. If you only know the show's name, search for it and read the `collectionId` off the result.