Can agents find and use your Actors?

Named task 0/2 passed

Haiku ×GPT Luna ×

1 / 9 passed

Date15161718192021

Discover↗↗↗✓↗↗↗

Named×××✓×××

Tasks and evidencecompass/google-maps-extractor ↗

Judge's suggested fix areas · 16 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection8
Output format8

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Get me 15 plumbers in Toronto with name, address, phone, website, and rating. Just the listings, no reviews. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used scrapesage/google-maps-scraper; expected compass/google-maps-extractor

Expected Actor not selected

Reported alternative: scrapesage/google-maps-scraper

namedActorFAIL: answer does not match /compass/(google-maps-extractor|crawler-google-places)/
rightActorFAIL: used scrapesage/google-maps-scraper; expected compass/google-maps-extractor

Judge's suggested fix area

Store search / Actor selection

check rightActor: used scrapesage/google-maps-scraper; expected compass/google-maps-extractor

Task ID

google-maps-extractor-discovery-toronto-plumbers

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used aolmo/google-maps-places; expected compass/google-maps-extractor

Expected Actor not selected

Reported alternative: aolmo/google-maps-places

namedActorFAIL: answer does not match /compass/(google-maps-extractor|crawler-google-places)/
rightActorFAIL: used aolmo/google-maps-places; expected compass/google-maps-extractor

Judge's suggested fix area

Store search / Actor selection

check rightActor: used aolmo/google-maps-places; expected compass/google-maps-extractor

Task ID

google-maps-extractor-discovery-toronto-plumbers

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used scrapesage/google-maps-scraper; expected compass/google-maps-extractor	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used aolmo/google-maps-places; expected compass/google-maps-extractor	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used scrapesage/google-maps-scraper; expected compass/google-maps-extractor	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used buff_pineapple/google-maps-scraper; expected compass/google-maps-extractor	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used scrapers_lat/canpages-canada-scraper, lulzasaur/houzz-scraper; expected compass/google-maps-extractor	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used scrapers_lat/overpass-osm-business-leads-scraper; expected compass/google-maps-extractor	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used gio21/plumber-scraper, aolmo/google-maps-places; expected compass/google-maps-extractor	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/google-maps-data-scraper; expected compass/google-maps-extractor	Open

Named task

0/2 passed2026-10-21

Using the compass/google-maps-extractor Actor, find 15 plumbers in Toronto and return name, address, phone number, and star rating for each. Say how many you got and name the highest-rated.

Failed Haiku claude-haiku-5.5

Observed result

check fields: 2 of 15 items missing address

fieldsFAIL: 2 of 15 items missing address

Judge's suggested fix area

Output format

check fields: 2 of 15 items missing address

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

google-maps-extractor-usage-toronto-plumbers

Failed GPT Luna gpt-6-luna

Observed result

check fields: 2 of 15 items missing address

fieldsFAIL: 2 of 15 items missing address

Judge's suggested fix area

Output format

check fields: 2 of 15 items missing address

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

google-maps-extractor-usage-toronto-plumbers

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check fields: 2 of 15 items missing address	Open
2026-10-21	GPT Luna	Failed	FAIL: check fields: 2 of 15 items missing address	Open
2026-10-20	Haiku	Failed	FAIL: check fields: 3 of 15 items missing address	Open
2026-10-20	GPT Luna	Failed	FAIL: check fields: 1 of 15 items missing address	Open
2026-10-19	Haiku	Failed	FAIL: check fields: 1 of 15 items missing address	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Failed	FAIL: check fields: 1 of 15 items missing address	Open
2026-10-16	Haiku	Failed	FAIL: check fields: 1 of 15 items missing address	Open
2026-10-15	Haiku	Failed	FAIL: check fields: 1 of 15 items missing address	Open

›

Contact Info ScrapergoogleTop suggested fix: Output format (9 of 17 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

1 / 9 passed
1 / 9 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

0 / 9 passed

Date15161718192021

Discover↗↗↗↗↗↗✓↗

Tasks and evidencevdrmota/contact-info-scraper ↗

Judge's suggested fix areas · 17 failed attempts over October 15 to 21, 2026, all models

Output format9
Store search / Actor selection8

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

find contact info for this website https://www.notion.so End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used delicious_zebu/contact-info-scraper; expected vdrmota/contact-info-scraper

Expected Actor not selected

Reported alternative: delicious_zebu/contact-info-scraper

namedActorFAIL: answer does not contain "vdrmota/contact-info-scraper"
rightActorFAIL: used delicious_zebu/contact-info-scraper; expected vdrmota/contact-info-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used delicious_zebu/contact-info-scraper; expected vdrmota/contact-info-scraper

Task ID

contact-info-scraper-discovery-notion-website

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used aitorsm/email-extractor; expected vdrmota/contact-info-scraper

Expected Actor not selected

Reported alternative: aitorsm/email-extractor

namedActorFAIL: answer does not contain "vdrmota/contact-info-scraper"
rightActorFAIL: used aitorsm/email-extractor; expected vdrmota/contact-info-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used aitorsm/email-extractor; expected vdrmota/contact-info-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

contact-info-scraper-discovery-notion-website

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/contact-info-scraper; expected vdrmota/contact-info-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used aitorsm/email-extractor; expected vdrmota/contact-info-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used emastra/website-contact-scraper; expected vdrmota/contact-info-scraper	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used gio21/website-contact-extractor; expected vdrmota/contact-info-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used gio21/website-contact-extractor; expected vdrmota/contact-info-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used jmlp/web-contact-scraper; expected vdrmota/contact-info-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used mikolabs/extract-emails-and-socials-contacts-any-website; expected vdrmota/contact-info-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used caprolok/website-email-phone-finder; expected vdrmota/contact-info-scraper	Open

Named task

0/2 passed2026-10-21

Using the vdrmota/contact-info-scraper Actor, get contact details from https://www.apify.com and https://www.notion.so without crawling deeper than one level. Report emails and social profiles per domain.

Failed Haiku claude-haiku-5.5

Observed result

check fields: 2 of 2 items missing url; 1 of 2 items missing emails

fieldsFAIL: 2 of 2 items missing url; 1 of 2 items missing emails

Judge's suggested fix area

Output format

check fields: 2 of 2 items missing url; 1 of 2 items missing emails

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

contact-info-scraper-usage-two-domains-shallow

Failed GPT Luna gpt-6-luna

Observed result

check fields: 2 of 2 items missing url; 1 of 2 items missing emails

fieldsFAIL: 2 of 2 items missing url; 1 of 2 items missing emails

Judge's suggested fix area

Output format

check fields: 2 of 2 items missing url; 1 of 2 items missing emails

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

contact-info-scraper-usage-two-domains-shallow

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-21	GPT Luna	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-20	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-20	GPT Luna	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-19	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-18	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-17	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-16	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open
2026-10-15	Haiku	Failed	FAIL: check fields: 2 of 2 items missing url; 1 of 2 items missing emails	Open

›

Google Maps With Contact DetailsgoogleTop suggested fix: Output format (8 of 13 failures)

Discovery 1/2 passed

Haiku ✓GPT Luna ↗

4 / 9 passed
4 / 9 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

0 / 8 passed

Date15161718192021

Discover✓↗↗↗↗✓✓↗

Tasks and evidencelukaskrivka/google-maps-with-contact-details ↗

Judge's suggested fix areas · 13 failed attempts over October 15 to 21, 2026, all models

Output format8
Store search / Actor selection5

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

1/2 passed2026-10-21

Find emails for 10 roofing companies in Toronto from Google Maps. Tell me how many had an email. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details

Expected Actor not selected

Reported alternative: buff_pineapple/google-maps-scraper

namedActorFAIL: answer does not contain "lukaskrivka/google-maps-with-contact-details"
rightActorFAIL: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details

Judge's suggested fix area

Store search / Actor selection

check rightActor: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details

Task ID

google-maps-contact-details-discovery-toronto-roofers-emails

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-maps-contact-details-discovery-toronto-roofers-emails

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used brainy_frostfield/roofing-lead-scraper-google-maps-phone-email; expected lukaskrivka/google-maps-with-contact-details	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used samstorm/roofing-lead-scraper; expected lukaskrivka/google-maps-with-contact-details	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used buff_pineapple/google-maps-scraper; expected lukaskrivka/google-maps-with-contact-details	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

0/2 passed2026-10-21

Using the lukaskrivka/google-maps-with-contact-details Actor, get 10 roofing companies in Toronto with name, website, email, and social profiles. Say how many had an email.

Failed Haiku claude-haiku-5.5

Observed result

check fields: 4 of 10 items missing emails

countFAIL (warn): answer does not match /[0-9]/
fieldsFAIL: 4 of 10 items missing emails
socialsFAIL (warn): lukaskrivka/google-maps-with-contact-details: scrapeSocialMediaProfiles is missing (expected to be set)

Judge's suggested fix area

Output format

check fields: 4 of 10 items missing emails

Task ID

google-maps-contact-details-usage-toronto-roofers

Failed GPT Luna gpt-6-luna

Observed result

check fields: 4 of 10 items missing emails

fieldsFAIL: 4 of 10 items missing emails
socialsFAIL (warn): lukaskrivka/google-maps-with-contact-details: scrapeSocialMediaProfiles is missing (expected to be set)

Judge's suggested fix area

Output format

check fields: 4 of 10 items missing emails

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

google-maps-contact-details-usage-toronto-roofers

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check fields: 4 of 10 items missing emails	Open
2026-10-21	GPT Luna	Failed	FAIL: check fields: 4 of 10 items missing emails	Open
2026-10-20	Haiku	Failed	FAIL: check fields: 5 of 10 items missing emails	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check fields: 3 of 10 items missing emails	Open
2026-10-18	Haiku	Failed	FAIL: check fields: 2 of 10 items missing emails	Open
2026-10-17	Haiku	Failed	FAIL: check fields: 2 of 10 items missing emails	Open
2026-10-16	Haiku	Failed	FAIL: check fields: 1 of 10 items missing emails	Open
2026-10-15	Haiku	Failed	FAIL: check fields: 2 of 10 items missing emails	Open

›

Booking ScrapergoogleTop suggested fix: Store search / Actor selection (9 of 11 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 9 passed
0 / 9 selected expected

See judge's finding (GPT Luna)

Named task 1/2 passed

Haiku ✓GPT Luna ×

6 / 8 passed

Date15161718192021

Named✓✓✓✓×✓✓×

Tasks and evidencevoyager/booking-scraper ↗

Judge's suggested fix areas · 11 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection9
Agent or model2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Get me 10 hotels in Amsterdam on Booking.com for two nights about a month from now. Which is cheapest? End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Expected Actor not selected

Reported alternative: santamaria-automations/booking-com-scraper

namedActorFAIL: answer does not contain "voyager/booking-scraper"
rightActorFAIL: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Task ID

booking-scraper-discovery-amsterdam-hotels

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Expected Actor not selected

Reported alternative: santamaria-automations/booking-com-scraper

namedActorFAIL: answer does not contain "voyager/booking-scraper"
rightActorFAIL: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

booking-scraper-discovery-amsterdam-hotels

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used santamaria-automations/booking-com-scraper; expected voyager/booking-scraper	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used rl1987/booking-api-scraper; expected voyager/booking-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used rl1987/booking-api-scraper; expected voyager/booking-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used rl1987/booking-api-scraper; expected voyager/booking-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used rl1987/booking-api-scraper; expected voyager/booking-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used rl1987/booking-api-scraper; expected voyager/booking-scraper	Open

Named task

1/2 passed2026-10-21

Using the voyager/booking-scraper Actor, find 10 hotels in Amsterdam for a two-night stay starting about four weeks from today, for three adults. Report how many you retrieved, the cheapest nightly price, and the highest-rated property.

Failed GPT Luna gpt-6-luna

Observed result

judge: task not completed

Judge's suggested fix area

Agent or model

The Actor produced 10 items, but after get-actor-run showed a lagging itemCount of 9 the agent stated 'nine hotels, not ten' and did not count the items returned by get-dataset-items (limit 20), so its count and extremes are wrong.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

booking-scraper-usage-amsterdam-three-adults

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

booking-scraper-usage-amsterdam-three-adults

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Failed	FAIL: judge: task not completed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	No result	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Crawler Google PlacesgoogleTop suggested fix: Store search / Actor selection (7 of 11 failures)

Discovery 0/1 passed

Haiku ↗GPT Luna ∕

0 / 7 passed
0 / 7 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

5 / 9 passed

Date15161718192021

Discover↗∕↗↗↗↗↗

Named×✓✓✓××✓×✓

Tasks and evidencecompass/crawler-google-places ↗

Judge's suggested fix areas · 11 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7
Output format3
Agent or model1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Track and compare the top 20 gyms in Miami by rating, review count, and opening hours. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used scrapesage/google-maps-scraper; expected compass/crawler-google-places

Expected Actor not selected

Reported alternative: scrapesage/google-maps-scraper

ranOkFAIL (warn): scrapesage/google-maps-scraper run : CALL-FAILED
namedActorFAIL: answer does not contain "compass/crawler-google-places"
rightActorFAIL: used scrapesage/google-maps-scraper; expected compass/crawler-google-places

Judge's suggested fix area

Store search / Actor selection

check rightActor: used scrapesage/google-maps-scraper; expected compass/crawler-google-places

Task ID

crawler-google-places-discovery-miami-gyms

Not counted GPT Luna gpt-6-luna

Observed result

session timed out

Selection not measured

ranOkFAIL (warn): scrapesage/google-maps-scraper run kwcVeUoWH8xhxorNK: RUNNING
namedActorFAIL: answer does not contain "compass/crawler-google-places"
rightActorFAIL: used scrapesage/google-maps-scraper; expected compass/crawler-google-places

Judge's suggested fix area

Not available

No additional diagnosis recorded.

Task ID

crawler-google-places-discovery-miami-gyms

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used scrapesage/google-maps-scraper; expected compass/crawler-google-places	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used scrapesage/google-maps-scraper; expected compass/crawler-google-places	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used scrapesage/google-maps-scraper; expected compass/crawler-google-places	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used datamech/apify-google-maps-scraper; expected compass/crawler-google-places	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used scrapeai/google-advanced-maps-places-scraper; expected compass/crawler-google-places	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used gio21/gym-scraper; expected compass/crawler-google-places	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used gio21/gym-scraper, scrapesage/google-maps-scraper; expected compass/crawler-google-places	Open

Named task

1/2 passed2026-10-21

Using the compass/crawler-google-places Actor, find 10 highly-rated coffee shops in Prague with name, address, phone number, and rating. Report how many you retrieved and name the highest-rated one.

Failed Haiku claude-haiku-5.5

Observed result

check items: 30 items, expected at most 10

itemsFAIL: 30 items, expected at most 10
capFAIL (warn): compass/crawler-google-places: maxCrawledPlacesPerSearch = 30 (expected at most 10)

Judge's suggested fix area

Output format

check items: 30 items, expected at most 10

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

crawler-google-places-usage-prague-coffee

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

crawler-google-places-usage-prague-coffee

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check items: 30 items, expected at most 10	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Failed	FAIL: check items: 30 items, expected at most 10	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Failed	FAIL: check items: 15 items, expected at most 10	Open

›

TripadvisorgoogleTop suggested fix: Store search / Actor selection (7 of 10 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 7 passed
0 / 7 selected expected

Named task Not counted

Haiku ∕GPT Luna ∕

2 / 5 passed

Test did not count

Date15161718192021

Discover↗↗∕↗∕↗↗

Named××✓✓×∕∕

Tasks and evidencemaxcopell/tripadvisor ↗

Judge's suggested fix areas · 10 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7
Agent or model2
Error messages1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Compare the top 10 attractions in Barcelona by rating and number of reviews, using TripAdvisor data. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper, axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Expected Actor not selected

Reported alternative: vulnv/tripadvisor-attractions-scraper

namedActorFAIL: answer does not match /maxcopell/tripadvisor(?!-)/
rightActorFAIL: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper, axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Judge's suggested fix area

Store search / Actor selection

check rightActor: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper, axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Task ID

tripadvisor-discovery-barcelona-attractions

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Expected Actor not selected

Reported alternative: axlymxp/tripadvisor-attractions-scraper

namedActorFAIL: answer does not match /maxcopell/tripadvisor(?!-)/
rightActorFAIL: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Judge's suggested fix area

Store search / Actor selection

check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

tripadvisor-discovery-barcelona-attractions

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper, axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used vulnv/tripadvisor-attractions-scraper, romy/tripadvisor-attractions-scraper, axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-19	Haiku	Not counted	No comment recorded	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used axlymxp/tripadvisor-attractions-scraper; expected maxcopell/tripadvisor	Open

Named task

0/2 passed2026-10-21

Using the maxcopell/tripadvisor Actor, find 10 top-rated restaurants in Rome with cuisine type, price range, and rating. Report how many you retrieved and name the top one with its rating.

Not counted Haiku claude-haiku-5.5

Observed result

session timed out

itemsFAIL: 0 items, expected at least 1
ranOkFAIL: maxcopell/tripadvisor run CNhj2AkDybmQbB7bP: RUNNING
shapeFAIL: answer does not match /\b[3-5]\.\d\b/

Judge's suggested fix area

None, nothing to fix

The Actor was healthy (0 failed requests, pages being crawled), but the agent kept polling get-actor-run until the timeout and never gave a final report; it did not budget time or finish with partial results.

Task ID

tripadvisor-usage-rome-restaurants

Not counted GPT Luna gpt-6-luna

Observed result

session timed out

itemsFAIL: 0 items, expected at least 1
ranOkFAIL: maxcopell/tripadvisor run TiJZgwGodQOYkRWhc: RUNNING
shapeFAIL: answer does not match /\b[3-5]\.\d\b/

Judge's suggested fix area

None, nothing to fix

The agent only repeated get-actor-run polls (runtime 280s+ with 0 dataset items and crawl stuck at 3/13 pages) and never checked partial results or adapted before the timeout, ending with a status message instead of an answer.

× 4 failed checks on GPT Luna

Task ID

tripadvisor-usage-rome-restaurants

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Not counted	No comment recorded	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Failed	FAIL: check ranOk: maxcopell/tripadvisor run pyzYytwa0UomvZiys: RUNNING	Open
2026-10-15	Haiku	Failed	FAIL: check grounded: 1 of 8 numbers not in Actor output: 2,700	Open

›

Google Search Scrapergoogle · 3 tasksTop suggested fix: Output format (2 of 3 failures)

Discovery Passed (2/2)

Haiku ✓GPT Luna ✓

9 / 9 passed
9 / 9 selected expected

Named task 0/1 passed

Haiku ∕GPT Luna ×

9 / 12 passed
2 named tasks

Date15161718192021

Named 1∕××✓∕∕∕

Named 2✓✓✓✓✓✓✓×

Tasks and evidenceapify/google-search-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Output format2
Input schema1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

None recorded on discovery tasks in this period.

Discovery task

Passed on all 22026-10-21

Scrape Google Search results (organic results, titles, links, snippets) for the query "apify web scraping", first results page only. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-search-scraper-discovery-organic-results

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-search-scraper-discovery-organic-results

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

0/2 passed2026-10-21

Using the apify/google-search-scraper Actor, get Google's AI Mode answer for "best crm for small business" in the UK. Report the answer text and how many sources it cited.

Not counted Haiku claude-haiku-5.5

Observed result

session timed out

ranOkFAIL: apify/google-search-scraper run IvbssE0ihAnjPPUrR: RUNNING

Judge's suggested fix area

None, nothing to fix

The run sat in RUNNING with statusMessage 'Nothing extracted yet' and itemCount 0 across polls from 44s to 230s of run time, with no progress signal that the agent could act on, and it hit the session timeout without producing data.

Task ID

google-search-scraper-usage-ai-mode-addon

Not counted GPT Luna gpt-6-luna

Observed result

session timed out

countryFAIL (warn): apify/google-search-scraper: countryCode = "GB" (expected "gb"); apify/google-search-scraper: countryCode is missing (expected "gb")
ranOkFAIL: apify/google-search-scraper run : CALL-FAILED; apify/google-search-scraper run QBh9c8stS2D4e5meu: RUNNING

Judge's suggested fix area

None, nothing to fix

The first call-actor with countryCode "GB" failed input validation, and the agent then abandoned countryCode, which the deterministic check flags as wrong or missing (expected "gb"). The schema's country field needs to make its lowercase enum clearer.

Task ID

google-search-scraper-usage-ai-mode-addon

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Not counted	No comment recorded	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Not counted	No comment recorded	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-16	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-15	Haiku	Not counted	No comment recorded	Open

Named task

1/2 passed2026-10-21

Using the apify/google-search-scraper Actor, get the first page of German Google results for "elektroauto leasing". Report how many organic results you got and the top three domains.

Failed GPT Luna gpt-6-luna

Observed result

check country: apify/google-search-scraper: countryCode is missing (expected "de")

langFAIL (warn): apify/google-search-scraper: languageCode is missing (expected "de")
countryFAIL: apify/google-search-scraper: countryCode is missing (expected "de")

Judge's suggested fix area

Input schema

check country: apify/google-search-scraper: countryCode is missing (expected "de")

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

google-search-scraper-usage-german-serp

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-search-scraper-usage-german-serp

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Failed	FAIL: check country: apify/google-search-scraper: countryCode is missing (expected "de")	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Google Maps Reviews ScrapergoogleTop suggested fix: Agent or model (1 of 1 failure)

Discovery Passed (2/2)

Haiku ✓GPT Luna ✓

9 / 9 passed
9 / 9 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

8 / 9 passed

Date15161718192021

Named✓✓✓×✓✓✓

Tasks and evidencecompass/google-maps-reviews-scraper ↗

Judge's suggested fix areas · 1 failed attempt over October 15 to 21, 2026, all models

Agent or model1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

None recorded on discovery tasks in this period.

Discovery task

Passed on all 22026-10-21

Summarize the main complaints and praises for the Ritz-Carlton in Chicago from its 30 most recent Google reviews. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-maps-reviews-discovery-ritz

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-maps-reviews-discovery-ritz

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

Passed on all 22026-10-21

Using the compass/google-maps-reviews-scraper Actor, pull the 30 most recent Google reviews of the Ritz-Carlton in Chicago, sorted newest first. Report the average star rating of the newest 15 versus the previous 15, and say whether the trend is up or down.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-maps-reviews-usage-ritz-trend

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

google-maps-reviews-usage-ritz-trend

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Socials 9 Actors

›

Facebook Groups ScrapersocialsTop suggested fixes: Agent or model, Store search / Actor selection (8 each of 16 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 8 passed
0 / 8 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

0 / 8 passed

Date15161718192021

Tasks and evidenceapify/facebook-groups-scraper ↗

Judge's suggested fix areas · 16 failed attempts over October 15 to 21, 2026, all models

Agent or model8
Store search / Actor selection8

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Get the 3 most recent posts from the public Facebook group at https://www.facebook.com/groups/874728723021553 and report what each post is about. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Expected Actor not selected

Reported alternative: memo23/facebook-public-group-posts-scraper

ranOkFAIL (warn): memo23/facebook-public-group-posts-scraper run : CALL-FAILED
namedActorFAIL: answer does not contain "apify/facebook-groups-scraper"
rightActorFAIL: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

facebook-groups-scraper-discovery-public-group-posts

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Expected Actor not selected

Reported alternative: memo23/facebook-public-group-posts-scraper

ranOkFAIL (warn): memo23/facebook-public-group-posts-scraper run : CALL-FAILED
namedActorFAIL: answer does not contain "apify/facebook-groups-scraper"
rightActorFAIL: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper

Task ID

facebook-groups-scraper-discovery-public-group-posts

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used apify/facebook-posts-scraper; expected apify/facebook-groups-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used curious_coder/facebook-post-scraper; expected apify/facebook-groups-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used apify/facebook-posts-scraper, scrapeforge/facebook-search-posts; expected apify/facebook-groups-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used memo23/facebook-public-group-posts-scraper; expected apify/facebook-groups-scraper	Open

Named task

0/2 passed2026-10-21

Using the apify/facebook-groups-scraper Actor, get 3 posts sorted by newest activity from the public Facebook group https://www.facebook.com/groups/874728723021553 and report each post's text and how many comments it has.

Failed Haiku claude-haiku-5.5

Observed result

check grounded: 4 of 17 numbers not in Actor output: 200, 290, 18, 280

Judge's suggested fix area

Agent or model

check grounded: 4 of 17 numbers not in Actor output: 200, 290, 18, 280

Task ID

facebook-groups-scraper-usage-recent-activity

Failed GPT Luna gpt-6-luna

Observed result

check grounded: 1 of 12 numbers not in Actor output: 18

Judge's suggested fix area

Agent or model

check grounded: 1 of 12 numbers not in Actor output: 18

Task ID

facebook-groups-scraper-usage-recent-activity

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check grounded: 4 of 17 numbers not in Actor output: 200, 290, 18, 280	Open
2026-10-21	GPT Luna	Failed	FAIL: check grounded: 1 of 12 numbers not in Actor output: 18	Open
2026-10-20	Haiku	Failed	FAIL: check grounded: 2 of 16 numbers not in Actor output: 200, 18	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check grounded: 2 of 9 numbers not in Actor output: 5, 200	Open
2026-10-18	Haiku	Failed	FAIL: check grounded: 2 of 9 numbers not in Actor output: 5, 300	Open
2026-10-17	Haiku	Failed	FAIL: check grounded: 2 of 10 numbers not in Actor output: 5, 11	Open
2026-10-16	Haiku	Failed	FAIL: check grounded: 1 of 7 numbers not in Actor output: 300	Open
2026-10-15	Haiku	Failed	FAIL: check grounded: 2 of 9 numbers not in Actor output: 400, 300	Open

›

Instagram Reel ScrapersocialsTop suggested fix: Output format (9 of 13 failures)

Discovery 0/2 passed

Haiku ×GPT Luna ↗

0 / 8 passed
7 / 8 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

3 / 8 passed

Date15161718192021

Discover×××××××

Named×✓✓×✓××

Tasks and evidenceapify/instagram-reel-scraper ↗

Judge's suggested fix areas · 13 failed attempts over October 15 to 21, 2026, all models

Output format9
Agent or model3
Store search / Actor selection1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

unseenuser/IG-posts1

Discovery task

0/2 passed2026-10-21

Get the 3 most recent Instagram Reels posted by @natgeo and report each reel's caption's first sentence, its play count and how many times the reels were shared. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

Expected Actor selected

Judge's suggested fix area

Output format

The call-actor result listed 'itemCount':2 and the field list had no shares field, while get-dataset-items returned 3 items with no share count even though includeSharesCount:true was set. The Actor's output did not deliver the promised shares data, and the run metadata did not match the dataset.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

instagram-reel-scraper-discovery-natgeo-reels-shares

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used unseenuser/IG-posts, intropix/instagram-posts-reels-scraper, data-slayer/instagram-post-details; expected apify/instagram-reel-scraper

Expected Actor not selected

Reported alternative: unseenuser/IG-posts

sharesFlagFAIL (warn): unseenuser/IG-posts: includeSharesCount is missing (expected true); intropix/instagram-posts-reels-scraper: includeSharesCount is missing (expected true); data-slayer/instagram-post-details: includeSharesCount is missing (expected true)
namedActorFAIL: answer does not contain "apify/instagram-reel-scraper"
rightActorFAIL: used unseenuser/IG-posts, intropix/instagram-posts-reels-scraper, data-slayer/instagram-post-details; expected apify/instagram-reel-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used unseenuser/IG-posts, intropix/instagram-posts-reels-scraper, data-slayer/instagram-post-details; expected apify/instagram-reel-scraper

Task ID

instagram-reel-scraper-discovery-natgeo-reels-shares

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used unseenuser/IG-posts, intropix/instagram-posts-reels-scraper, data-slayer/instagram-post-details; expected apify/instagram-reel-scraper	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-18	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-17	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-16	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-15	Haiku	Failed	FAIL: judge: task not completed	Open

Named task

0/2 passed2026-10-21

Using the apify/instagram-reel-scraper Actor, get the 3 most recent Reels from the Instagram profile @natgeo, skipping any pinned reels, and report each reel's caption's first sentence, play count, and duration in seconds.

Failed Haiku claude-haiku-5.5

Observed result

check items: 10 items, expected at most 3

itemsFAIL: 10 items, expected at most 3
threeReelsFAIL (warn): apify/instagram-reel-scraper: resultsLimit = 10 (expected at most 3)

Judge's suggested fix area

Output format

check items: 10 items, expected at most 3

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-reel-scraper-usage-natgeo-skip-pinned

Failed GPT Luna gpt-6-luna

Observed result

check grounded: 1 of 13 numbers not in Actor output: 18

Judge's suggested fix area

Agent or model

check grounded: 1 of 13 numbers not in Actor output: 18

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

× 4 failed checks on Haiku

Task ID

instagram-reel-scraper-usage-natgeo-skip-pinned

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check items: 10 items, expected at most 3	Open
2026-10-21	GPT Luna	Failed	FAIL: check grounded: 1 of 13 numbers not in Actor output: 18	Open
2026-10-20	Haiku	Failed	FAIL: check grounded: 1 of 15 numbers not in Actor output: 22	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Failed	FAIL: check grounded: 1 of 14 numbers not in Actor output: 8.9	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Failed	FAIL: check items: 10 items, expected at most 3	Open

›

Instagram ScrapersocialsTop suggested fix: Store search / Actor selection (7 of 12 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 7 passed
0 / 7 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

3 / 8 passed

Date15161718192021

Discover↗↗↗↗↗∕↗

Named✓✓×✓×××

Tasks and evidenceapify/instagram-scraper ↗

Judge's suggested fix areas · 12 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7
Agent or model5

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

apify/instagram-post-scraper ours7

Discovery task

0/2 passed2026-10-21

Get the 3 most recent posts from the Instagram profile @nasa. When you are done, reply with exactly two lines: Actor: <username/name of the Apify Actor you used> Posts: <number of posts you retrieved>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper

Expected Actor not selected

Reported alternative: apify/instagram-post-scraper

schemaValiditycall-actor: data/waitSecs must be <= 45
ranOkFAIL (warn): apify/instagram-post-scraper run : CALL-FAILED
namedActorFAIL: answer does not contain "apify/instagram-scraper"
rightActorFAIL: used apify/instagram-post-scraper; expected apify/instagram-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper

Task ID

instagram-scraper-discovery-nasa-posts

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper

Expected Actor not selected

Reported alternative: apify/instagram-post-scraper

namedActorFAIL: answer does not contain "apify/instagram-scraper"
rightActorFAIL: used apify/instagram-post-scraper; expected apify/instagram-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper

Task ID

instagram-scraper-discovery-nasa-posts

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-20	Haiku	Not counted	No comment recorded	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used apify/instagram-post-scraper; expected apify/instagram-scraper	Open

Named task

0/2 passed2026-10-21

How engaged is the Instagram creator @nasa's audience compared to their follower count? Use the apify/instagram-scraper Actor to get their profile and their recent posts, then answer with: the follower count, the average likes per recent post, and one sentence interpreting the engagement level.

Failed Haiku claude-haiku-5.5

Observed result

check grounded: 1 of 4 numbers not in Actor output: 528,120

Judge's suggested fix area

Agent or model

check grounded: 1 of 4 numbers not in Actor output: 528,120

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-scraper-usage-engagement-nasa

Failed GPT Luna gpt-6-luna

Observed result

check grounded: 1 of 2 numbers not in Actor output: 454,114

Judge's suggested fix area

Agent or model

check grounded: 1 of 2 numbers not in Actor output: 454,114

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-scraper-usage-engagement-nasa

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check grounded: 1 of 4 numbers not in Actor output: 528,120	Open
2026-10-21	GPT Luna	Failed	FAIL: check grounded: 1 of 2 numbers not in Actor output: 454,114	Open
2026-10-20	Haiku	Failed	FAIL: check grounded: 2 of 3 numbers not in Actor output: 648,800, 523,000	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check grounded: 2 of 3 numbers not in Actor output: 410,929, 7,398,725	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Failed	FAIL: check grounded: 1 of 2 numbers not in Actor output: 351,163	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Instagram Hashtag ScrapersocialsTop suggested fix: Output format (4 of 8 failures)

Discovery 1/2 passed

Haiku ✓GPT Luna ↗

5 / 7 passed
5 / 7 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

2 / 8 passed

Date15161718192021

Discover✓↗∕✓✓✓✓↗

Named×××✓×✓×

Tasks and evidenceapify/instagram-hashtag-scraper ↗

Judge's suggested fix areas · 8 failed attempts over October 15 to 21, 2026, all models

Output format4
Agent or model2
Store search / Actor selection2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

1/2 passed2026-10-21

Find 5 Instagram posts tagged #nasa and report which accounts posted them and what the posts are about. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used scraping_solutions/instagram-hashtag-scraper-pro-no-cookies; expected apify/instagram-hashtag-scraper

Expected Actor not selected

Reported alternative: scraping_solutions/instagram-hashtag-scraper-pro-no-cookies

namedActorFAIL: answer does not contain "apify/instagram-hashtag-scraper"
rightActorFAIL: used scraping_solutions/instagram-hashtag-scraper-pro-no-cookies; expected apify/instagram-hashtag-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used scraping_solutions/instagram-hashtag-scraper-pro-no-cookies; expected apify/instagram-hashtag-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-hashtag-scraper-discovery-nasa-hashtag

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

instagram-hashtag-scraper-discovery-nasa-hashtag

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used scraping_solutions/instagram-hashtag-scraper-pro-no-cookies; expected apify/instagram-hashtag-scraper	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used data-slayer/instagram-hashtag-posts-reels-scraper-no-login; expected apify/instagram-hashtag-scraper	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

0/2 passed2026-10-21

Using the apify/instagram-hashtag-scraper Actor, get 5 Instagram reels based on keyword nasa and report each post's caption's first sentence and its like count.

Failed Haiku claude-haiku-5.5

Observed result

check items: 6 items, expected at most 5

itemsFAIL: 6 items, expected at most 5
keywordModeFAIL (warn): apify/instagram-hashtag-scraper: keywordSearch = false (expected true)

Judge's suggested fix area

Output format

check items: 6 items, expected at most 5; check grounded: 1 of 12 numbers not in Actor output: 2

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-hashtag-scraper-usage-nasa-keyword-reels

Failed GPT Luna gpt-6-luna

Observed result

check items: 6 items, expected at most 5

itemsFAIL: 6 items, expected at most 5
keywordModeFAIL (warn): apify/instagram-hashtag-scraper: keywordSearch = false (expected true)

Judge's suggested fix area

Output format

check items: 6 items, expected at most 5

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-hashtag-scraper-usage-nasa-keyword-reels

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check items: 6 items, expected at most 5	Open
2026-10-21	GPT Luna	Failed	FAIL: check items: 6 items, expected at most 5	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check items: 6 items, expected at most 5	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Failed	FAIL: check grounded: 5 of 6 numbers not in Actor output: 1, 2, 3, 7.5, 0	Open
2026-10-16	Haiku	Failed	FAIL: check items: 6 items, expected at most 5	Open
2026-10-15	Haiku	Failed	FAIL: check grounded: 1 of 12 numbers not in Actor output: 12,000	Open

›

Zillow Zip SearchsocialsTop suggested fix: Store search / Actor selection (8 of 14 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 8 passed
0 / 8 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

3 / 9 passed

Date15161718192021

Named✓×××××✓×✓

Tasks and evidencemaxcopell/zillow-zip-search ↗

Judge's suggested fix areas · 14 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection8
Input schema6

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Find 10 homes currently for sale in ZIP code 85251 (Scottsdale, Arizona) priced under $800,000, with address and price for each. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used epctex/realtor-scraper; expected ^maxcopell/zillow

Expected Actor not selected

Reported alternative: epctex/realtor-scraper

namedActorFAIL: answer does not contain "maxcopell/zillow"
rightActorFAIL: used epctex/realtor-scraper; expected ^maxcopell/zillow

Judge's suggested fix area

Store search / Actor selection

check rightActor: used epctex/realtor-scraper; expected ^maxcopell/zillow

Task ID

zillow-zip-search-discovery-scottsdale

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used alizarin_refrigerator-owner/zillow-scraper, bgfc97/zillow-real-estate-scraper; expected ^maxcopell/zillow

Expected Actor not selected

Reported alternative: alizarin_refrigerator-owner/zillow-scraper

namedActorFAIL: answer does not contain "maxcopell/zillow"
rightActorFAIL: used alizarin_refrigerator-owner/zillow-scraper, bgfc97/zillow-real-estate-scraper; expected ^maxcopell/zillow

Judge's suggested fix area

Store search / Actor selection

check rightActor: used alizarin_refrigerator-owner/zillow-scraper, bgfc97/zillow-real-estate-scraper; expected ^maxcopell/zillow

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

zillow-zip-search-discovery-scottsdale

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used epctex/realtor-scraper; expected ^maxcopell/zillow	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used alizarin_refrigerator-owner/zillow-scraper, bgfc97/zillow-real-estate-scraper; expected ^maxcopell/zillow	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used epctex/realtor-scraper; expected ^maxcopell/zillow	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used nextgensda/zillow-listings; expected ^maxcopell/zillow	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used charitable_trousseau/zillow-search-scraper; expected ^maxcopell/zillow	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used blackfalcondata/realtor-scraper; expected ^maxcopell/zillow	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used pear_fight/zillow-scraper; expected ^maxcopell/zillow	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used bgfc97/zillow-real-estate-scraper; expected ^maxcopell/zillow	Open

Named task

1/2 passed2026-10-21

Using the maxcopell/zillow-zip-search Actor, find homes for sale in ZIP code 85251 with a maximum price of $800,000, limited to 10 results. Report how many you retrieved and the cheapest one's address and price.

Failed Haiku claude-haiku-5.5

Observed result

check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)

forSaleByOwnerTrueFAIL: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)

Judge's suggested fix area

Input schema

check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

zillow-zip-search-usage-scottsdale

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

zillow-zip-search-usage-scottsdale

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-18	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-17	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-16	Haiku	Failed	FAIL: check forSaleByOwnerTrue: maxcopell/zillow-zip-search: forSaleByOwner = false (expected true)	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Facebook Ads ScrapersocialsTop suggested fix: Store search / Actor selection (7 of 10 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 7 passed
0 / 7 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

5 / 8 passed

Date15161718192021

Discover∕↗↗↗↗↗↗

Named✓✓✓×××✓

Tasks and evidenceapify/facebook-ads-scraper ↗

Judge's suggested fix areas · 10 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7
Agent or model2
Output format1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

What Facebook and Instagram ads is Nike currently running, according to Meta's public Ad Library? Look up ads for facebook.com/nike, limit it to 10 ads, and report how many active ads you found and one example body text from them. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used curious_coder/facebook-ads-library-scraper; expected apify/facebook-ads-scraper

Expected Actor not selected

Reported alternative: curious_coder/facebook-ads-library-scraper

namedActorFAIL: answer does not contain "apify/facebook-ads-scraper"
rightActorFAIL: used curious_coder/facebook-ads-library-scraper; expected apify/facebook-ads-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used curious_coder/facebook-ads-library-scraper; expected apify/facebook-ads-scraper

Task ID

facebook-ads-scraper-discovery-nike-ad-library

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used affiliatescope/meta-ads-scraper; expected apify/facebook-ads-scraper

Expected Actor not selected

Reported alternative: affiliatescope/meta-ads-scraper

ranOkFAIL (warn): affiliatescope/meta-ads-scraper run : CALL-FAILED
namedActorFAIL: answer does not contain "apify/facebook-ads-scraper"
rightActorFAIL: used affiliatescope/meta-ads-scraper; expected apify/facebook-ads-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used affiliatescope/meta-ads-scraper; expected apify/facebook-ads-scraper

Task ID

facebook-ads-scraper-discovery-nike-ad-library

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used curious_coder/facebook-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used affiliatescope/meta-ads-scraper; expected apify/facebook-ads-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used curious_coder/facebook-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used adside/meta-ad-library-ads-scraper, nourishing_courier/meta-ads-scraper-pro, hyperbach/meta-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used adside/meta-ad-library-ads-scraper, wakeful_outhouse/facebook-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used adside/meta-ad-library-ads-scraper, hyperbach/meta-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used adside/meta-ad-library-ads-scraper, hyperbach/meta-ads-library-scraper; expected apify/facebook-ads-scraper	Open
2026-10-15	Haiku	Not counted	No comment recorded	Open

Named task

Passed on all 22026-10-21

Using the apify/facebook-ads-scraper Actor, get the ads from the last 30 days in Meta's Ad Library for the Facebook page https://www.facebook.com/nike, limited to 20 results. Include the reach and spend data. Report how many ads you actually retrieved and one real ad's body text with reach data.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

facebook-ads-scraper-usage-nike-last-30-days

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

facebook-ads-scraper-usage-nike-last-30-days

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check grounded: 1 of 5 numbers not in Actor output: 16	Open
2026-10-18	Haiku	Failed	FAIL: check grounded: 1 of 4 numbers not in Actor output: 19	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Instagram Profile ScrapersocialsTop suggested fix: Output format (3 of 3 failures)

Discovery Passed (2/2)

Haiku ✓GPT Luna ✓

8 / 8 passed
8 / 8 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

5 / 8 passed

Date15161718192021

Named✓✓✓✓✓××

Tasks and evidenceapify/instagram-profile-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Output format3

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

None recorded on discovery tasks in this period.

Discovery task

Passed on all 22026-10-21

Is the Instagram account @natgeo verified, and what does its bio say? End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

instagram-profile-scraper-discovery-natgeo-verified

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

instagram-profile-scraper-discovery-natgeo-verified

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

0/2 passed2026-10-21

Using the apify/instagram-profile-scraper Actor, look up the Instagram profile @natgeo and report its bio's first sentence, whether it's a business account and which country the account is from.

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

Judge's suggested fix area

Output format

The run had includeAboutSection=true, yet the dataset item from get-dataset-items had no country or about fields, and the run's field list also omitted them, so the output did not match what the input schema promises. The agent therefore could not report the country.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

instagram-profile-scraper-usage-natgeo-about-section

Failed GPT Luna gpt-6-luna

Observed result

judge: task not completed

Judge's suggested fix area

Output format

With includeAboutSection=true the run's dataset exposed 26 fields, none of them country or about-section fields, so the agent could not report a country; the Actor's documented country-of-origin output does not appear in the dataset schema.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

instagram-profile-scraper-usage-natgeo-about-section

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Failed	FAIL: judge: task not completed	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Facebook Posts ScrapersocialsTop suggested fix: Store search / Actor selection (2 of 3 failures)

Discovery 1/2 passed

Haiku ↗GPT Luna ✓

5 / 7 passed
5 / 7 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

7 / 8 passed

Date15161718192021

Discover✓✓∕✓✓↗↗✓

Named✓✓✓✓×✓✓

Tasks and evidenceapify/facebook-posts-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection2
Agent or model1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

thedoor/facebook-page-scraper2

Discovery task

1/2 passed2026-10-21

Get the 5 most recent posts from NASA's Facebook page (facebook.com/NASA) with each post's text and reaction count. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used thedoor/facebook-page-scraper; expected apify/facebook-posts-scraper

Expected Actor not selected

Reported alternative: thedoor/facebook-page-scraper

namedActorFAIL: answer does not contain "apify/facebook-posts-scraper"
rightActorFAIL: used thedoor/facebook-page-scraper; expected apify/facebook-posts-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used thedoor/facebook-page-scraper; expected apify/facebook-posts-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

facebook-posts-scraper-discovery-nasa-page

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

facebook-posts-scraper-discovery-nasa-page

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used thedoor/facebook-page-scraper; expected apify/facebook-posts-scraper	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used thedoor/facebook-page-scraper; expected apify/facebook-posts-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

Passed on all 22026-10-21

Using the apify/facebook-posts-scraper Actor, get the 5 most recent posts from https://www.facebook.com/NASA and report each post's first sentence and its reactions count.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

facebook-posts-scraper-usage-nasa-engagement

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

facebook-posts-scraper-usage-nasa-engagement

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check grounded: 4 of 5 numbers not in Actor output: 223,607, 1,401, 1,462, 223,000	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Instagram Post ScrapersocialsTop suggested fix: Agent or model (3 of 3 failures)

Discovery Passed (2/2)

Haiku ✓GPT Luna ✓

8 / 8 passed
8 / 8 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

4 / 7 passed

Date15161718192021

Named×✓✓✓×∕×✓

Tasks and evidenceapify/instagram-post-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Agent or model3

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

None recorded on discovery tasks in this period.

Discovery task

Passed on all 22026-10-21

Get the most recent post from the Instagram profile @natgeo and report its caption's first sentence and how many comments it has. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

instagram-post-scraper-discovery-natgeo-latest-post

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

instagram-post-scraper-discovery-natgeo-latest-post

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

1/2 passed2026-10-21

Using the apify/instagram-post-scraper Actor, get the 3 most recent posts from the Instagram profile @natgeo, skipping pinned posts, and report each post's caption's first sentence and its like count.

Failed Haiku claude-haiku-5.5

Observed result

check grounded: 2 of 13 numbers not in Actor output: 22, 18

Judge's suggested fix area

Agent or model

check grounded: 2 of 13 numbers not in Actor output: 22, 18

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

instagram-post-scraper-usage-natgeo-three-posts

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

× 4 failed checks on Haiku

Task ID

instagram-post-scraper-usage-natgeo-three-posts

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check grounded: 2 of 13 numbers not in Actor output: 22, 18	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Not counted	No comment recorded	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check grounded: 1 of 7 numbers not in Actor output: 2	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Failed	FAIL: check grounded: 1 of 11 numbers not in Actor output: 2	Open

Universe 5 Actors

›

Cheerio ScraperuniverseTop suggested fix: Error messages (7 of 12 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

1 / 6 passed
1 / 6 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

0 / 7 passed

Date15161718192021

Discover✓∕∕↗↗↗↗

Named×∕×××××

Tasks and evidenceapify/cheerio-scraper ↗

Judge's suggested fix areas · 12 failed attempts over October 15 to 21, 2026, all models

Error messages7
Store search / Actor selection5

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

I need product names and prices from IKEA's chest of drawers category https://www.ikea.com/gb/en/cat/chest-of-drawers-drawer-units-st004/. It's a large catalog page and I don't need JavaScript-rendered content, just what's in the server-rendered HTML, so I want something fast and lightweight rather than a full headless browser, since I'll be running this often. Run it against that first category page only and show me 10 results so I can check the extraction looks right before I scale it up. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper

Expected Actor not selected

Reported alternative: pintostudio/ikea-products-bycategory

pageCapFAIL (warn): pintostudio/ikea-products-bycategory: maxPagesPerCrawl is missing (expected at most 5); pintostudio/ikea-products-bycategory: maxPagesPerCrawl is missing (expected at most 5)
pageFunctionFAIL: pintostudio/ikea-products-bycategory: pageFunction is missing (expected to be set); pintostudio/ikea-products-bycategory: pageFunction is missing (expected to be set)
namedActorFAIL: answer does not contain "apify/cheerio-scraper"
rightActorFAIL: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper

Task ID

cheerio-scraper-discovery-ikea-static-catalog

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used apify/beautifulsoup-scraper; expected apify/cheerio-scraper

Expected Actor not selected

Reported alternative: apify/beautifulsoup-scraper

pageCapFAIL (warn): apify/beautifulsoup-scraper: maxPagesPerCrawl is missing (expected at most 5)
namedActorFAIL: answer does not contain "apify/cheerio-scraper"
rightActorFAIL: used apify/beautifulsoup-scraper; expected apify/cheerio-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used apify/beautifulsoup-scraper; expected apify/cheerio-scraper

Task ID

cheerio-scraper-discovery-ikea-static-catalog

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used apify/beautifulsoup-scraper; expected apify/cheerio-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used pintostudio/ikea-products-bycategory; expected apify/cheerio-scraper	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

0/2 passed2026-10-21

Run the apify/cheerio-scraper Actor on https://www.foyles.co.uk/category. From there follow the link to the manga category and get the books listed in it. Stop after 2 hops of following links and don't let it process more than 20 pages total. For each book page, get me the title and the price as a JSON object with title and price keys, and show me 5 of them.

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

itemsFAIL (warn): 0 items, expected at least 1

Judge's suggested fix area

Error messages

The Actor run reported SUCCEEDED with 'Total 1 requests: 0 succeeded, 1 failed' and the dataset held only a '#error' item with 'Request blocked - received 403 status code', so the agent had no actionable guidance such as a proxy hint to get past the block.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

cheerio-scraper-usage-foyles-manga-two-hops

Failed GPT Luna gpt-6-luna

Observed result

check ranOk: 3 Actor run(s), expected at most 2

itemsFAIL (warn): 0 items, expected at least 1
ranOkFAIL: 3 Actor run(s), expected at most 2

Judge's suggested fix area

Error messages

check ranOk: 3 Actor run(s), expected at most 2

× 4 failed checks on Haiku

Task ID

cheerio-scraper-usage-foyles-manga-two-hops

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Failed	FAIL: check ranOk: 3 Actor run(s), expected at most 2	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check ranOk: 5 Actor run(s), expected at most 2	Open
2026-10-18	Haiku	Failed	FAIL: check ranOk: 6 Actor run(s), expected at most 2	Open
2026-10-17	Haiku	Failed	FAIL: check ranOk: apify/cheerio-scraper run : CALL-FAILED	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Failed	FAIL: check ranOk: 5 Actor run(s), expected at most 2	Open

›

RAG Web BrowseruniverseTop suggested fix: Store search / Actor selection (7 of 10 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 8 passed
1 / 8 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

6 / 8 passed

Date15161718192021

Discover↗↗↗↗↗×↗

Named✓✓✓✓×✓×✓

Tasks and evidenceapify/rag-web-browser ↗

Judge's suggested fix areas · 10 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7
Agent or model2
Input schema1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

apify/ai-web-scraper ours1
apify/google-search-scraper ours1
No Actor reported5

Discovery task

0/2 passed2026-10-21

Search the web for the 5 most popular James Bond movies by box office and give me the answer as clean text with the pages you used as sources. Don't answer from memory; fetch the live pages and quote them. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser

Expected Actor not selected

Reported alternative: none

queryFAIL: no Actor run with a recorded input
ranOkFAIL (warn): 0 Actor run(s), expected at least 1
namedActorFAIL: answer does not contain "apify/rag-web-browser"
rightActorFAIL: no Actor run and no matching tool call; expected apify/rag-web-browser

Judge's suggested fix area

Store search / Actor selection

check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser

Task ID

rag-web-browser-discovery-web-search-with-sources

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used apify/google-search-scraper, smart-digital/website-content-text-extractor; expected apify/rag-web-browser

Expected Actor not selected

Reported alternative: apify/google-search-scraper

queryFAIL: apify/google-search-scraper: query is missing (expected to be set); smart-digital/website-content-text-extractor: query is missing (expected to be set)
namedActorFAIL: answer does not contain "apify/rag-web-browser"
rightActorFAIL: used apify/google-search-scraper, smart-digital/website-content-text-extractor; expected apify/rag-web-browser

Judge's suggested fix area

Store search / Actor selection

check rightActor: used apify/google-search-scraper, smart-digital/website-content-text-extractor; expected apify/rag-web-browser

Task ID

rag-web-browser-discovery-web-search-with-sources

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used apify/google-search-scraper, smart-digital/website-content-text-extractor; expected apify/rag-web-browser	Open
2026-10-20	Haiku	Failed	FAIL: check query: no Actor run with a recorded input	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used apify/ai-web-scraper; expected apify/rag-web-browser	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: no Actor run and no matching tool call; expected apify/rag-web-browser	Open

Named task

1/2 passed2026-10-21

Using the apify/rag-web-browser Actor, search Google for "fastest growing startups in San Francisco 2026" and return the top 5 results as Markdown, with each page's URL and a two-line summary of what it says.

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

Judge's suggested fix area

Agent or model

The Actor run succeeded with 5 items, but the agent's final message after the output-token cutoff resumed mid-sentence, so the delivered answer lacks items 1-3 and its header. Summaries also exceed the two-line format. This is an agent output-length and format failure, not an Actor problem.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

rag-web-browser-usage-sf-startups-markdown

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

itemsFAIL (warn): 10 items, expected at most 5

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

× 4 failed checks on GPT Luna

Task ID

rag-web-browser-usage-sf-startups-markdown

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check sources: answer does not match /https?:///	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Website Content CrawleruniverseTop suggested fixes: Store search / Actor selection, Error messages (4 each of 12 failures)

Discovery 1/2 passed

Haiku ✓GPT Luna ↗

2 / 6 passed
2 / 6 selected expected

Named task 0/2 passed

Haiku ×GPT Luna ×

0 / 8 passed

Date15161718192021

Discover∕↗∕↗↗✓✓↗

Tasks and evidenceapify/website-content-crawler ↗

Judge's suggested fix areas · 12 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection4
Error messages4
Agent or model2
Input schema2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

1/2 passed2026-10-21

I want to build a knowledge base from a documentation website so I can later use it for RAG and AI search. Can you crawl the docs at https://docs.apify.com/academy and extract clean text or Markdown content from each page? Limit it to 20 pages so it doesn't run forever, and make sure whatever tool you use is meant for turning websites into LLM-ready text, not just retrieving raw HTML. Report how many pages you got and the titles of five of them. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used s3nafps/ai-ready-webpage-extractor; expected apify/website-content-crawler

Expected Actor not selected

Reported alternative: s3nafps/ai-ready-webpage-extractor

pageCapFAIL (warn): s3nafps/ai-ready-webpage-extractor: maxCrawlPages is missing (expected at most 20)
startUrlFAIL: s3nafps/ai-ready-webpage-extractor: startUrls[0].url is missing (expected to match /docs\.apify\.com/academy/)
namedActorFAIL: answer does not contain "apify/website-content-crawler"
rightActorFAIL: used s3nafps/ai-ready-webpage-extractor; expected apify/website-content-crawler

Judge's suggested fix area

Store search / Actor selection

check rightActor: used s3nafps/ai-ready-webpage-extractor; expected apify/website-content-crawler

Task ID

website-content-crawler-discovery-docs-knowledge-base

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

website-content-crawler-discovery-docs-knowledge-base

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used s3nafps/ai-ready-webpage-extractor; expected apify/website-content-crawler	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used precious_bathmat/website-to-markdown; expected apify/website-content-crawler	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used maximedupre/webpage-text-extractor; expected apify/website-content-crawler	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used sauliusautomatesit/website-content-crawler-api; expected apify/website-content-crawler	Open
2026-10-15	Haiku	Not counted	No comment recorded	Open

Named task

0/2 passed2026-10-21

Use the apify/website-content-crawler Actor to crawl only the platform section of the Apify docs, https://docs.apify.com/platform, and nothing outside it. Cap it at 10 pages. I need the clean text of each page and also the full HTML of the pages you scraped. Report how many pages you got and confirm the HTML was saved.

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

itemsFAIL (warn): 11 items, expected at most 10
saveHtmlFAIL (warn): apify/website-content-crawler: saveHtml is missing (expected true); apify/website-content-crawler: saveHtml is missing (expected true)
sectionFilterFAIL (warn): apify/website-content-crawler: includeUrlGlobs is missing (expected to be set)

Judge's suggested fix area

Agent or model

The Actor ran fine, but the agent used non-schema keys (saveHtmlAsFile, maxResults), so saveHtml was missing. After the glob rerun returned 1 item, it spent its remaining turns on curl probing, ran out of turns, and gave no final report.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

website-content-crawler-usage-platform-section-html

Failed GPT Luna gpt-6-luna

Observed result

check ranOk: apify/website-content-crawler run : CALL-FAILED

saveHtmlFAIL (warn): apify/website-content-crawler: saveHtml is missing (expected true); apify/website-content-crawler: saveHtml is missing (expected true)
ranOkFAIL: apify/website-content-crawler run : CALL-FAILED

Judge's suggested fix area

Error messages

check ranOk: apify/website-content-crawler run : CALL-FAILED

× 5 failed checks on Haiku

Task ID

website-content-crawler-usage-platform-section-html

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Failed	FAIL: check ranOk: apify/website-content-crawler run : CALL-FAILED	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check ranOk: 3 Actor run(s), expected at most 2	Open
2026-10-18	Haiku	Failed	FAIL: check ranOk: 3 Actor run(s), expected at most 2	Open
2026-10-17	Haiku	Failed	FAIL: check ranOk: 5 Actor run(s), expected at most 2	Open
2026-10-16	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-15	Haiku	Failed	FAIL: judge: task not completed	Open

›

AI Web ScraperuniverseTop suggested fix: Store search / Actor selection (4 of 5 failures)

Discovery 0/1 passed

Haiku ↗GPT Luna ∕

0 / 4 passed
0 / 4 selected expected

Named task 1/2 passed

Haiku ✓GPT Luna ×

7 / 8 passed

Date15161718192021

Discover∕∕∕↗↗↗↗

Named✓✓✓✓✓✓✓×

Tasks and evidenceapify/ai-web-scraper ↗

Judge's suggested fix areas · 5 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection4
Error messages1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

I need recent articles about climate change from these 3 news sites, and I don't want to write any selectors or code, just describe what I need. From each site take up to 5 articles with article title, author name, and publication date. Sites are https://www.theguardian.com/environment/climate-crisis, https://www.bbc.com/news/science-environment and https://www.reuters.com/sustainability/climate-energy/. Report how many articles you got per site and show three examples. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used automation-lab/news-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper

Expected Actor not selected

Reported alternative: automation-lab/news-article-extractor

pageCapFAIL (warn): automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20); automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20); automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20); moving_beacon-owner1/my-actor-62: maxPagesToVisit is missing (expected at most 20); automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20)
promptGivenFAIL (warn): automation-lab/news-article-extractor: prompt is missing (expected to be set); automation-lab/news-article-extractor: prompt is missing (expected to be set); automation-lab/news-article-extractor: prompt is missing (expected to be set); moving_beacon-owner1/my-actor-62: prompt is missing (expected to be set); automation-lab/news-article-extractor: prompt is missing (expected to be set)
ranOkFAIL (warn): 5 Actor run(s), expected at most 3
namedActorFAIL: answer does not contain "apify/ai-web-scraper"
rightActorFAIL: used automation-lab/news-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used automation-lab/news-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper

Task ID

ai-web-scraper-discovery-climate-articles-three-sites

Not counted GPT Luna gpt-6-luna

Observed result

Actor run Epvz5Q5EnnCVnUnnN (xmolodtsov/google-news-scraper) FAILED

Selection not measured

pageCapFAIL (warn): automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20); automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20); scrapesage/google-news-scraper: maxPagesToVisit is missing (expected at most 20); xmolodtsov/google-news-scraper: maxPagesToVisit is missing (expected at most 20); scrapesage/google-news-scraper: maxPagesToVisit is missing (expected at most 20); automation-lab/news-article-extractor: maxPagesToVisit is missing (expected at most 20)
promptGivenFAIL (warn): automation-lab/news-article-extractor: prompt is missing (expected to be set); automation-lab/news-article-extractor: prompt is missing (expected to be set); scrapesage/google-news-scraper: prompt is missing (expected to be set); xmolodtsov/google-news-scraper: prompt is missing (expected to be set); scrapesage/google-news-scraper: prompt is missing (expected to be set); automation-lab/news-article-extractor: prompt is missing (expected to be set)
ranOkFAIL (warn): 6 Actor run(s), expected at most 3
namedActorFAIL: answer does not contain "apify/ai-web-scraper"
rightActorFAIL: used automation-lab/news-article-extractor, scrapesage/google-news-scraper, xmolodtsov/google-news-scraper; expected apify/ai-web-scraper

Judge's suggested fix area

None, nothing to fix

The search-actors call with keywords 'news articles' surfaced news scrapers such as automation-lab/news-article-extractor and the Google News scrapers, not apify/ai-web-scraper. The agent then chained runs of the wrong actors, so a no-selector, describe-what-you-want extractor needs to rank higher for this kind of request.

Task ID

ai-web-scraper-discovery-climate-articles-three-sites

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used automation-lab/news-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used automation-lab/news-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used scrapepilot/rss-feed-scraper----news-scraper-article-extractor, moving_beacon-owner1/my-actor-62; expected apify/ai-web-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used scrapepilot/rss-feed-scraper----news-scraper-article-extractor, automation-lab/news-article-extractor; expected apify/ai-web-scraper	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Not counted	No comment recorded	Open

Named task

1/2 passed2026-10-21

Using the apify/ai-web-scraper Actor, get the Actors listed on the first page of https://apify.com/store and give me each Actor's name, description, rating and number of users in a table. One page only, no crawling.

Failed GPT Luna gpt-6-luna

Observed result

check ranOk: 6 Actor run(s), expected at most 2

onePageFAIL (warn): apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3); apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3); apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3); apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3); apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3); apify/ai-web-scraper: maxPagesToVisit is missing (expected at most 3)
ranOkFAIL: 6 Actor run(s), expected at most 2

Judge's suggested fix area

Error messages

check ranOk: 6 Actor run(s), expected at most 2

Task ID

ai-web-scraper-usage-apify-store-first-page

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

ai-web-scraper-usage-apify-store-first-page

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Failed	FAIL: check ranOk: 6 Actor run(s), expected at most 2	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Web ScraperuniverseTop suggested fix: Store search / Actor selection (3 of 5 failures)

Discovery Not counted

Haiku ∕GPT Luna ∕

1 / 4 passed
1 / 4 selected expected

Test did not count

Named task 1/2 passed

Haiku ✓GPT Luna ×

1 / 3 passed

Date15161718192021

Discover↗∕∕↗↗✓∕

Named∕∕∕∕∕×✓×

Tasks and evidenceapify/web-scraper ↗

Judge's suggested fix areas · 5 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection3
Error messages2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

I need product names and prices from https://www.waterstones.com/category/fiction/science-fiction-fantasy-horror. It's a large catalog site that needs JavaScript execution, so make sure to use a real browser. Run it against the first category page only and show me 10 results so I can check the extraction looks right before I scale it up. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Not counted Haiku claude-haiku-5.5

Observed result

session timed out

Selection not measured

namedActorFAIL: answer does not contain "apify/web-scraper"
rightActorFAIL: used apify/playwright-scraper; expected apify/web-scraper

Judge's suggested fix area

None, nothing to fix

Searches for 'browser scraper' and 'playwright scraper' never surfaced apify/web-scraper, so the agent settled on apify/playwright-scraper (rightActor FAIL), and its run then timed out with 0 items.

Task ID

web-scraper-discovery-waterstones-headless

Not counted GPT Luna gpt-6-luna

Observed result

Actor run jxsrQTDihdT6Qvhtx (eszetael_lab/reliable-playwright-scraper) FAILED

Selection not measured

schemaValiditycall-actor: data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties; call-actor: data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have additional properties, data must NOT have a
pageCapFAIL (warn): eszetael_lab/reliable-playwright-scraper: maxPagesPerCrawl is missing (expected at most 5); eszetael_lab/reliable-playwright-scraper: maxPagesPerCrawl is missing (expected at most 5)
ranOkFAIL (warn): eszetael_lab/reliable-playwright-scraper run jxsrQTDihdT6Qvhtx: FAILED
namedActorFAIL: answer does not contain "apify/web-scraper"
rightActorFAIL: used eszetael_lab/reliable-playwright-scraper; expected apify/web-scraper

Judge's suggested fix area

None, nothing to fix

The search-actors call with 'website browser scrape' (limit 5) returned a job-extractor Actor, and the agent then picked eszetael_lab/reliable-playwright-scraper without ever surfacing apify/web-scraper.

Task ID

web-scraper-discovery-waterstones-headless

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used parseforge/e-commerce-scraper, apify/jsdom-scraper; expected apify/web-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used apify/puppeteer-scraper, apify/rag-web-browser; expected apify/web-scraper	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used webdatatools/css-selector-extractor-js; expected apify/web-scraper	Open

Named task

1/2 passed2026-10-21

Using the apify/web-scraper Actor, scrape https://coinmarketcap.com/ and give me the names and current prices of the top 10 cryptocurrencies by market cap listed there. One page only.

Failed GPT Luna gpt-6-luna

Observed result

check ranOk: 3 Actor run(s), expected at most 2

pageCapFAIL (warn): apify/web-scraper: maxPagesPerCrawl is missing (expected at most 3); apify/web-scraper: maxPagesPerCrawl is missing (expected at most 3); apify/web-scraper: maxPagesPerCrawl is missing (expected at most 3)
ranOkFAIL: 3 Actor run(s), expected at most 2

Judge's suggested fix area

Error messages

check ranOk: 3 Actor run(s), expected at most 2

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

web-scraper-usage-coinmarketcap-top-ten

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

web-scraper-usage-coinmarketcap-top-ten

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Failed	FAIL: check ranOk: 3 Actor run(s), expected at most 2	Open
2026-10-20	Haiku	Failed	FAIL: check ranOk: 3 Actor run(s), expected at most 2	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Not counted	No comment recorded	Open
2026-10-18	Haiku	Not counted	No comment recorded	Open
2026-10-17	Haiku	Not counted	No comment recorded	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Not counted	No comment recorded	Open

Video 8 Actors

›

Indeed ScrapervideoTop suggested fix: Store search / Actor selection (8 of 9 failures)

Discovery 0/2 passed

Haiku ↗GPT Luna ↗

0 / 8 passed
0 / 8 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

7 / 8 passed

Date15161718192021

Named✓✓✓✓✓✓×✓

Tasks and evidencemisceres/indeed-scraper ↗

Judge's suggested fix areas · 9 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection8
Agent or model1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

I am hiring a marketing coordinator in Denver, CO. Show me 10 similar postings on Indeed with salary and company ratings so I can benchmark our offer. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used curious_coder/indeed-scraper; expected misceres/indeed-scraper

Expected Actor not selected

Reported alternative: curious_coder/indeed-scraper

namedActorFAIL: answer does not contain "misceres/indeed-scraper"
rightActorFAIL: used curious_coder/indeed-scraper; expected misceres/indeed-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used curious_coder/indeed-scraper; expected misceres/indeed-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

indeed-scraper-discovery-denver-marketing

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper

Expected Actor not selected

Reported alternative: kaix/indeed-scraper

namedActorFAIL: answer does not contain "misceres/indeed-scraper"
rightActorFAIL: used kaix/indeed-scraper; expected misceres/indeed-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

indeed-scraper-discovery-denver-marketing

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used curious_coder/indeed-scraper; expected misceres/indeed-scraper	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used valig/indeed-jobs-scraper; expected misceres/indeed-scraper	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used valig/indeed-jobs-scraper; expected misceres/indeed-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used borderline/indeed-scraper; expected misceres/indeed-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used kaix/indeed-scraper; expected misceres/indeed-scraper	Open

Named task

1/2 passed2026-10-21

Using the misceres/indeed-scraper Actor, pull 20 current software engineer job postings with their full descriptions, and report the 5 skills mentioned most often across those descriptions.

Failed Haiku claude-haiku-5.5

Observed result

capFAIL (warn): misceres/indeed-scraper: maxItems is missing (expected at most 20)
shapeFAIL: answer does not match /(python|aws|sql|java|docker|react|kubernetes)/

Judge's suggested fix area

Agent or model

Task ID

indeed-scraper-usage-swe-skills

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

capFAIL (warn): misceres/indeed-scraper: maxItems is missing (expected at most 20)

Judge's suggested fix area

Agent or model

The Actor did its part (SUCCEEDED, 20 items with descriptions). The weakness is the agent's redundant re-fetching of the same descriptions in many 5-item and 1-item get-dataset-items calls (offsets 1,2,3,6,7,8,12,13,16,17), after it already had the first 20-item fetch.

Task ID

indeed-scraper-usage-swe-skills

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: check shape: answer does not match /(python\|aws\|sql\|java\|docker\|react\|kubernetes)/	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Amazon Reviews ScrapervideoTop suggested fix: Error messages (6 of 11 failures)

Discovery 0/1 passed

Haiku ∕GPT Luna ×

1 / 6 passed
3 / 6 selected expected

Named task 1/2 passed

Haiku ×GPT Luna ✓

2 / 8 passed

Date15161718192021

Discover×∕↗↗↗✓×

Named×××××✓×✓

Tasks and evidencejunglee/amazon-reviews-scraper ↗

Judge's suggested fix areas · 11 failed attempts over October 15 to 21, 2026, all models

Error messages6
Store search / Actor selection3
Agent or model2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Get the 30 most helpful reviews for this Amazon product and tell me the top complaints and praises: https://www.amazon.com/dp/B0CWXNS552 End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Failed GPT Luna gpt-6-luna

Observed result

check namedActor: answer does not contain "junglee/amazon-reviews-scraper"

Expected Actor selected

ranOkFAIL (warn): 4 Actor run(s), expected at most 3
namedActorFAIL: answer does not contain "junglee/amazon-reviews-scraper"

Judge's suggested fix area

Agent or model

check namedActor: answer does not contain "junglee/amazon-reviews-scraper"

Task ID

amazon-reviews-scraper-discovery-product-complaints

Not counted Haiku claude-haiku-5.5

Observed result

Actor run vNQGlpoFKTbFuN8QL (junglee/amazon-reviews-scraper) ABORTED

Selection not measured

ranOkFAIL (warn): junglee/amazon-reviews-scraper run vNQGlpoFKTbFuN8QL: ABORTED

Judge's suggested fix area

None, nothing to fix

The run sat at 'Crawled 5/6 pages, 0 failed requests' with 10 items across several get-actor-run polls and gave no sign why it had stalled, so the agent had to abort it (ABORTED) and ended up with 10 of 30 reviews.

Task ID

amazon-reviews-scraper-discovery-product-complaints

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Failed	FAIL: check namedActor: answer does not contain "junglee/amazon-reviews-scraper"	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/amazon-reviews-scraper-with-advanced-filters; expected junglee/amazon-reviews-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used igview-owner/amazon-review-scraper; expected junglee/amazon-reviews-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/amazon-reviews-scraper-with-advanced-filters; expected junglee/amazon-reviews-scraper	Open
2026-10-16	Haiku	Not counted	No comment recorded	Open
2026-10-15	Haiku	Failed	FAIL: judge: task not completed	Open

Named task

1/2 passed2026-10-21

Using the junglee/amazon-reviews-scraper Actor, pull the 30 most helpful reviews for https://www.amazon.com/dp/B0CWXNS552 and summarize the top 3 complaints and top 3 praises.

Failed Haiku claude-haiku-5.5

Observed result

judge: task not completed

Judge's suggested fix area

Agent or model

The Actor ran fine, but the agent's final answer starts mid-sentence ('needed a keyring or holder...') after an output token limit cut, dropping complaints 1-2 and the 10-of-30 caveat from the final message; the agent should keep its output short enough to deliver a complete final answer.

Check/judge disagreement. Hard checks: passed. Task-completion rubric: failed or unavailable.

Task ID

amazon-reviews-scraper-usage-helpful-thirty

Passed GPT Luna gpt-6-luna · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

amazon-reviews-scraper-usage-helpful-thirty

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check ranOk: junglee/amazon-reviews-scraper run : CALL-FAILED	Open
2026-10-18	Haiku	Failed	FAIL: check ranOk: junglee/amazon-reviews-scraper run : CALL-FAILED	Open
2026-10-17	Haiku	Failed	FAIL: check ranOk: junglee/amazon-reviews-scraper run : CALL-FAILED	Open
2026-10-16	Haiku	Failed	FAIL: check ranOk: junglee/amazon-reviews-scraper run : CALL-FAILED	Open
2026-10-15	Haiku	Failed	FAIL: check ranOk: junglee/amazon-reviews-scraper run : CALL-FAILED	Open

›

TikTok Profile ScrapervideoTop suggested fix: Store search / Actor selection (3 of 3 failures)

Discovery 1/2 passed

Haiku ✓GPT Luna ↗

6 / 9 passed
6 / 9 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

9 / 9 passed

Date15161718192021

Discover✓✓✓✓✓↗✓↗

Named✓✓✓✓✓✓✓

Tasks and evidenceclockworks/tiktok-profile-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection3

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

1/2 passed2026-10-21

How many followers does @apifylife have on TikTok, and what are their 5 most recent videos? End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor GPT Luna gpt-6-luna

Observed result

check rightActor: used scrapingmonkey/tiktok-scraper, novi/tiktok-user-api; expected clockworks/tiktok-profile-scraper

Expected Actor not selected

Reported alternative: scrapingmonkey/tiktok-scraper

namedActorFAIL: answer does not contain "clockworks/tiktok-profile-scraper"
rightActorFAIL: used scrapingmonkey/tiktok-scraper, novi/tiktok-user-api; expected clockworks/tiktok-profile-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used scrapingmonkey/tiktok-scraper, novi/tiktok-user-api; expected clockworks/tiktok-profile-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

tiktok-profile-scraper-discovery-apifylife-followers

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Expected Actor selected

schemaValiditycall-actor: data/waitSecs must be <= 45
ranOkFAIL (warn): clockworks/tiktok-profile-scraper run : CALL-FAILED

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

tiktok-profile-scraper-discovery-apifylife-followers

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Other Actor	FAIL: check rightActor: used scrapingmonkey/tiktok-scraper, novi/tiktok-user-api; expected clockworks/tiktok-profile-scraper	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-scraper; expected clockworks/tiktok-profile-scraper	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used scrapingmonkey/tiktok-scraper, novi/tiktok-user-api, igview-owner/tiktok-profile-video-scraper; expected clockworks/tiktok-profile-scraper	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

Passed on all 22026-10-21

Using the clockworks/tiktok-profile-scraper Actor, scrape the TikTok profile @apifylife and report its follower count plus the 5 most recent videos with their play counts.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

itemsFAIL (warn): 10 items, expected at most 5

Judge's suggested fix area

Input schema

The first call-actor run with resultsPerPage:5 returned two pinned 2024 videos, so only 3 genuine recent videos came back; the pinned-posts behavior forced a second run (10 items total, 'items' check FAIL). The excludePinnedPosts description and default should make this clearer so a 'most recent N' request is right on the first run.

Task ID

tiktok-profile-scraper-usage-apifylife-latest-five

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

itemsFAIL (warn): 10 items, expected at most 5

Judge's suggested fix area

Input schema

The first call-actor used excludePinnedPosts:false and returned two old pinned videos (2024) among the 5 results, forcing a second run (yKm3m6z4oHuxqVPqf) and leaving 10 items in total; the schema or docs should make the pinned-post behavior with 'latest' sorting clearer.

Task ID

tiktok-profile-scraper-usage-apifylife-latest-five

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

TikTok ScrapervideoTop suggested fix: Store search / Actor selection (8 of 12 failures)

Discovery 0/1 passed

Haiku ↗GPT Luna ∕

0 / 8 passed
0 / 8 selected expected

Named task Not counted

Haiku ∕GPT Luna ∕

3 / 7 passed

Test did not count

Date15161718192021

Named✓×××✓✓×∕

Tasks and evidenceclockworks/tiktok-scraper ↗

Judge's suggested fix areas · 12 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection8
Agent or model2
Error messages2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Pull the top 20 TikTok videos under the hashtag #cleantok and summarize what content is trending and how much engagement it gets. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used clockworks/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper

Expected Actor not selected

Reported alternative: clockworks/tiktok-hashtag-scraper

namedActorFAIL: answer does not contain "clockworks/tiktok-scraper"
rightActorFAIL: used clockworks/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used clockworks/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper

Task ID

tiktok-scraper-discovery-cleantok

Not counted GPT Luna gpt-6-luna

Observed result

MCP server exposed no tools to the agent; MCP server apify failed

Selection not measured

ranOkFAIL (warn): 0 Actor run(s), expected at least 1
namedActorFAIL: answer does not contain "clockworks/tiktok-scraper"
rightActorFAIL: no Actor run and no matching tool call; expected clockworks/tiktok-scraper

Judge's suggested fix area

None, nothing to fix

The MCP server apify failed and exposed no tools, so the agent never searched the store or ran clockworks/tiktok-scraper. It also produced an empty final answer, with no explanation of the failure and no 'Actor:' line.

Task ID

tiktok-scraper-discovery-cleantok

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used clockworks/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used clockworks/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used memo23/tiktok-hashtag-scraper; expected clockworks/tiktok-scraper	Open

Named task

0/2 passed2026-10-21

Using the clockworks/tiktok-scraper Actor, search TikTok for "protein coffee recipe" videos and report the 10 with the most plays, including their play counts.

Not counted Haiku claude-haiku-5.5

Observed result

MCP server exposed no tools to the agent; MCP server apify failed

playsFAIL (warn): no Actor run to read items from
ranOkFAIL: 0 Actor run(s), expected at least 1
rightActorFAIL: no Actor run and no matching tool call; expected clockworks/tiktok-scraper
shapeFAIL: answer does not match /\d{4,}|\d{1,3}(,\d{3})+/

Judge's suggested fix area

None, nothing to fix

Infrastructure facts show 'MCP server apify failed' and 'exposed no tools', so the agent's only Bash call found no APIFY_TOKEN or CLI and could not run clockworks/tiktok-scraper; the MCP connection failure gave no actionable error for the agent to recover from.

Task ID

tiktok-scraper-usage-protein-coffee

Not counted GPT Luna gpt-6-luna

Observed result

MCP server exposed no tools to the agent; MCP server apify failed

playsFAIL (warn): no Actor run to read items from
ranOkFAIL: 0 Actor run(s), expected at least 1
rightActorFAIL: no Actor run and no matching tool call; expected clockworks/tiktok-scraper
shapeFAIL: answer does not match /\d{4,}|\d{1,3}(,\d{3})+/

Judge's suggested fix area

None, nothing to fix

The infrastructure facts say 'MCP server apify failed' and 'MCP server exposed no tools to the agent', so the agent could not call clockworks/tiktok-scraper and gave no answer or explanation. The MCP connection failure should be surfaced clearly so it can be fixed.

× 3 failed checks on GPT Luna

Task ID

tiktok-scraper-usage-protein-coffee

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Not counted	No comment recorded	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Failed	FAIL: check ranOk: clockworks/tiktok-scraper run : CALL-FAILED	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-17	Haiku	Failed	FAIL: check grounded: 1 of 10 numbers not in Actor output: 499,100	Open
2026-10-16	Haiku	Failed	FAIL: check ranOk: clockworks/tiktok-scraper run ErgokAj0dZezslLLq: RUNNING	Open
2026-10-15	Haiku	Passed	Passed	Open

›

Amazon CrawlervideoTop suggested fix: Error messages (5 of 8 failures)

Discovery 0/1 passed

Haiku ∕GPT Luna ×

4 / 7 passed
6 / 7 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

3 / 8 passed

Date15161718192021

Discover✓✓✓✓↗××

Named×××××✓✓

Tasks and evidencejunglee/amazon-crawler ↗

Judge's suggested fix areas · 8 failed attempts over October 15 to 21, 2026, all models

Error messages5
Agent or model2
Store search / Actor selection1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

powerai/amazon-product-search-scraper1

Discovery task

0/2 passed2026-10-21

Find the top 10 best-selling wireless earbuds on Amazon with their prices and ratings. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Failed GPT Luna gpt-6-luna

Observed result

check namedActor: answer does not contain "junglee/amazon-crawler"

Expected Actor selected

shapeFAIL: answer does not match /\d+[.,]\d{2}/
ranOkFAIL (warn): 4 Actor run(s), expected at most 3
namedActorFAIL: answer does not contain "junglee/amazon-crawler"

Judge's suggested fix area

Agent or model

check namedActor: answer does not contain "junglee/amazon-crawler"; check shape: answer does not match /\d+[.,]\d{2}/

Task ID

amazon-crawler-discovery-wireless-earbuds

Not counted Haiku claude-haiku-5.5

Observed result

Actor run moyy9mhbIlvvb0vgQ (junglee/Amazon-crawler) FAILED

Selection not measured

ranOkFAIL (warn): junglee/Amazon-crawler run moyy9mhbIlvvb0vgQ: FAILED

Judge's suggested fix area

None, nothing to fix

The first run's get-dataset-items output (qJ3o7bMeYcW2hfQyv) returned rows with missing price.value (HAOYUYAN, B0HCTKSSQK), null stars (motorcycle earbuds), and three duplicate TOZO A1 variant rows, so the agent could not produce a clean top-10 with prices and ratings.

Task ID

amazon-crawler-discovery-wireless-earbuds

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Not counted	No comment recorded	Open
2026-10-21	GPT Luna	Failed	FAIL: check namedActor: answer does not contain "junglee/amazon-crawler"	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used powerai/amazon-product-search-scraper, automly/amazon-products-scraper---fast-efficient-with-sales-data; expected junglee/amazon-crawler	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

Passed on all 22026-10-21

Using the junglee/amazon-crawler Actor, scrape https://www.amazon.com/s?k=wireless+earbuds and report the top 10 products with price and rating.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

fieldFAIL (warn): 1 of 10 items fail price exists undefined

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

amazon-crawler-usage-earbuds-search-url

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

fieldFAIL (warn): 2 of 10 items fail price exists undefined

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

amazon-crawler-usage-earbuds-search-url

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Failed	FAIL: check ranOk: junglee/amazon-crawler run : CALL-FAILED	Open
2026-10-18	Haiku	Failed	FAIL: check ranOk: junglee/amazon-crawler run : CALL-FAILED	Open
2026-10-17	Haiku	Failed	FAIL: check ranOk: junglee/amazon-crawler run : CALL-FAILED	Open
2026-10-16	Haiku	Failed	FAIL: check ranOk: junglee/amazon-crawler run : CALL-FAILED	Open
2026-10-15	Haiku	Failed	FAIL: check ranOk: junglee/amazon-crawler run : CALL-FAILED	Open

›

YouTube ScrapervideoTop suggested fix: Store search / Actor selection (7 of 7 failures)

Discovery 0/1 passed

Haiku ↗GPT Luna ?

1 / 8 passed
1 / 8 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

9 / 9 passed

Date15161718192021

Discover↗↗↗✓↗↗↗

Named✓✓✓✓✓✓✓

Tasks and evidencestreamers/youtube-scraper ↗

Judge's suggested fix areas · 7 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection7

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

Find the 10 most-viewed YouTube videos about home espresso machines published in the last month, with view counts and channel names. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper

Expected Actor not selected

Reported alternative: api-ninja/youtube-search-scraper

namedActorFAIL: answer does not contain "streamers/youtube-scraper"
rightActorFAIL: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper

Check/judge disagreement. Hard checks: failed. Task-completion rubric: passed.

Task ID

youtube-scraper-discovery-espresso

No result GPT Luna gpt-6-luna

Observed result

No result is available for this date.

Selection not measured

namedActorFAIL: answer does not contain "streamers/youtube-scraper"
rightActorFAIL: used memo23/youtube-search-results-scraper; expected streamers/youtube-scraper

Judge's suggested fix area

Not available

No additional diagnosis recorded.

Task ID

youtube-scraper-discovery-espresso

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper	Open
2026-10-21	GPT Luna	No result	No comment recorded	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper	Open
2026-10-20	GPT Luna	Other Actor	FAIL: check rightActor: used memo23/youtube-search-results-scraper; expected streamers/youtube-scraper	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/youtube-video-scraper-by-keyword; expected streamers/youtube-scraper	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Other Actor	FAIL: check rightActor: used api-ninja/youtube-search-scraper; expected streamers/youtube-scraper	Open
2026-10-16	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/youtube-video-scraper-by-keyword, api-ninja/youtube-search-scraper; expected streamers/youtube-scraper	Open
2026-10-15	Haiku	Other Actor	FAIL: check rightActor: used delicious_zebu/youtube-video-scraper-by-keyword; expected streamers/youtube-scraper	Open

Named task

Passed on all 22026-10-21

Using the streamers/youtube-scraper Actor, get the top 5 YouTube search results for "home espresso machine review" and report each video's views, likes, and comment count.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

Agent or model

The get-dataset-items result showed order values 0-4, with Best of 2026 at order 0, yet the final table ranks it #4 and calls the list "sorted by rank". This is the agent's own reasoning error, not an Actor failure.

Task ID

youtube-scraper-usage-espresso-engagement

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

youtube-scraper-usage-espresso-engagement

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

YouTube Shorts ScrapervideoTop suggested fix: Store search / Actor selection (4 of 6 failures)

Discovery 0/1 passed

Haiku ↗GPT Luna ?

4 / 8 passed
4 / 8 selected expected

Named task Passed (1/1)

Haiku ✓GPT Luna ?

6 / 8 passed

Date15161718192021

Discover✓✓✓↗↗↗✓↗

Named✓××✓✓✓✓

Tasks and evidencestreamers/youtube-shorts-scraper ↗

Judge's suggested fix areas · 6 failed attempts over October 15 to 21, 2026, all models

Store search / Actor selection4
Agent or model2

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

Discovery task

0/2 passed2026-10-21

What are the 10 most popular Shorts on the MrBeast YouTube channel? End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Other Actor Haiku claude-haiku-5.5

Observed result

check rightActor: used easyapi/youtube-channel-shorts-scraper; expected streamers/youtube-shorts-scraper

Expected Actor not selected

Reported alternative: easyapi/youtube-channel-shorts-scraper

namedActorFAIL: answer does not contain "streamers/youtube-shorts-scraper"
rightActorFAIL: used easyapi/youtube-channel-shorts-scraper; expected streamers/youtube-shorts-scraper

Judge's suggested fix area

Store search / Actor selection

check rightActor: used easyapi/youtube-channel-shorts-scraper; expected streamers/youtube-shorts-scraper

Task ID

youtube-shorts-scraper-discovery-mrbeast-popular

No result GPT Luna gpt-6-luna

Observed result

No result is available for this date.

Selection not measured

namedActorFAIL: answer does not contain "streamers/youtube-shorts-scraper"
rightActorFAIL: used datagleaner/youtube-channel-scraper; expected streamers/youtube-shorts-scraper

Judge's suggested fix area

Not available

No additional diagnosis recorded.

Task ID

youtube-shorts-scraper-discovery-mrbeast-popular

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Other Actor	FAIL: check rightActor: used easyapi/youtube-channel-shorts-scraper; expected streamers/youtube-shorts-scraper	Open
2026-10-21	GPT Luna	No result	No comment recorded	Open
2026-10-20	Haiku	Other Actor	FAIL: check rightActor: used easyapi/youtube-channel-shorts-scraper; expected streamers/youtube-shorts-scraper	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Other Actor	FAIL: check rightActor: used streamers/youtube-scraper; expected streamers/youtube-shorts-scraper	Open
2026-10-18	Haiku	Other Actor	FAIL: check rightActor: used grow_media/youtube-channel-video-scraper; expected streamers/youtube-shorts-scraper	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

1/2 passed2026-10-21

Using the streamers/youtube-shorts-scraper Actor, get the 10 most popular Shorts from the MrBeast channel and report each one's view and like counts.

No result GPT Luna gpt-6-luna

Observed result

No result is available for this date.

Judge's suggested fix area

Not available

No additional diagnosis recorded.

Task ID

youtube-shorts-scraper-usage-mrbeast-top-ten

Passed Haiku claude-haiku-5.5 · Open full session ↗

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

youtube-shorts-scraper-usage-mrbeast-top-ten

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	No result	No comment recorded	Open
2026-10-20	Haiku	Passed	Passed	Open
2026-10-20	GPT Luna	Passed	Passed	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-16	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-15	Haiku	Passed	Passed	Open

›

TikTok Comments ScrapervideoTop suggested fix: Agent or model (2 of 3 failures)

Discovery Passed (2/2)

Haiku ✓GPT Luna ✓

7 / 8 passed
8 / 8 selected expected

Named task Passed (2/2)

Haiku ✓GPT Luna ✓

6 / 8 passed

Date15161718192021

Discover✓✓✓✓✓×✓

Named××✓✓✓✓✓

Tasks and evidenceclockworks/tiktok-comments-scraper ↗

Judge's suggested fix areas · 3 failed attempts over October 15 to 21, 2026, all models

Agent or model2
Error messages1

Diagnoses, not proven causes. Read the evidence before changing the Actor.

Alternatives the agent reported instead

None recorded on discovery tasks in this period.

Discovery task

Passed on all 22026-10-21

Here's a TikTok video: https://www.tiktok.com/@apifylife/video/7332342275151760642. What are people saying in the comments? List the 5 most-liked ones. End your reply with exactly one line: Actor: <username/name of the Apify Actor you used>

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

tiktok-comments-scraper-discovery-video-top-comments

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Expected Actor selected

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

tiktok-comments-scraper-discovery-video-top-comments

All 7 daily results for this task, every model

Date	Model	Outcome	Evidence	Session
2026-10-21	Haiku	Passed	Passed	Open
2026-10-21	GPT Luna	Passed	Passed	Open
2026-10-20	Haiku	Failed	FAIL: judge: task not completed	Open
2026-10-20	GPT Luna	Not counted	No comment recorded	Open
2026-10-19	Haiku	Passed	Passed	Open
2026-10-18	Haiku	Passed	Passed	Open
2026-10-17	Haiku	Passed	Passed	Open
2026-10-16	Haiku	Passed	Passed	Open
2026-10-15	Haiku	Passed	Passed	Open

Named task

Passed on all 22026-10-21

Using the clockworks/tiktok-comments-scraper Actor, get the top 20 comments (no replies) from this TikTok video: https://www.tiktok.com/@apifylife/video/7332342275151760642, and report the 5 comments with the most likes.

Passed Haiku claude-haiku-5.5

Observed result

The task passed; checks and judge agree.

capFAIL (warn): clockworks/tiktok-comments-scraper: commentsPerPost is missing (expected at most 20)

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.

Task ID

tiktok-comments-scraper-usage-top-level-only

Passed GPT Luna gpt-6-luna

Observed result

The task passed; checks and judge agree.

Judge's suggested fix area

None, nothing to fix

Nothing to add; the task passed.