Introducing the ‘preferred’ livecrawl option that tries to fetch fresh content but gracefully falls back to cached results when crawling fails, providing the best of both worlds.
Date: 7 June 2025
We’ve added a new livecrawl
option called "preferred"
that provides a more resilient approach to content fetching. This option attempts to crawl fresh content but gracefully falls back to cached results when live crawling fails.
The preferred
option is now available in both /contents
and /search_and_contents
endpoints.
The new livecrawl: "preferred"
option provides intelligent fallback behavior:
The key difference between "preferred"
and "always"
:
Option | Crawl Fails + Cache Available | Crawl Fails + No Cache |
---|---|---|
"preferred" | Returns cached content | Returns crawl error |
"always" | Returns crawl error | Returns crawl error |
This makes "preferred"
more resilient for production applications where you want fresh content when possible, but don’t want requests to fail when websites are temporarily unavailable.
If content freshness is critical and you want nothing else, then using "always"
might be better.
The "preferred"
option is ideal when:
Here are all four livecrawl options and their behaviors:
Option | Crawl Behavior | Cache Fallback | Best For |
---|---|---|---|
"always" | Always crawls | Never falls back | Critical real-time data, willing to accept failures |
"preferred" | Always crawls | Falls back on crawl failure | Fresh content with reliability |
"fallback" | Only if no cache | Uses cache first | Balanced speed and freshness |
"never" | Never crawls | Always uses cache | Maximum speed |
If you’re currently using livecrawl: "always"
but experiencing reliability issues:
This change maintains your preference for fresh content while improving reliability.