Docs: explain how sessions interact with retries (keeping cookies/session across a retry)

[vdusek]

Summary

The Session management guide doesn't explain how sessions interact with retries, nor how to keep a session (and its cookies) across a retry. This is a recurring source of confusion.

Motivation

See discussion #2028: a user sets cookies via send_request (or establishes login state), raises to trigger a retry, and finds the cookies "gone" on the retry.

The cookies aren't lost, they're stored on the session. The catch is that an unpinned request draws a new random session on every attempt. When the handler raises and the request is retried, the crawler picks a fresh session from the pool (default max_pool_size=1000), so the retry runs on a different, cookie-less session. Nothing in the docs sets this expectation, and the natural mental model ("same request → same session") is wrong by default.

Proposal

Add a short section to the Session management guide, e.g. "Reusing a session across retries", covering the default behavior and the ways to keep a session across retries.

Option 1 — single-session pool

The simplest fix, keeps a raise-to-retry flow unchanged. One session for everything, so cookies set on the first attempt are still there on the retry. Trade-off: no session rotation.

from crawlee.crawlers import HttpCrawler
from crawlee.sessions import SessionPool


async def main() -> None:
    crawler = HttpCrawler(session_pool=SessionPool(max_pool_size=1))
    # ...

Option 2 — pin the request to its session

Keeps rotation for everything else. Instead of raising, re-enqueue the same URL bound to the session that now has the cookies. The user_data flag guards against re-enqueuing in a loop, and always_enqueue=True bypasses deduplication so the same URL runs again.

from crawlee import Request
from crawlee.crawlers import HttpCrawler, HttpCrawlingContext

crawler = HttpCrawler()


@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
    if context.request.user_data.get('prepared'):
        # Cookies from the setup step are on this pinned session.
        # ... do the real work ...
        return

    # First pass: establish cookies on the current session.
    await context.send_request('https://example.com/set-something')

    # Re-enqueue the same URL, pinned to the same session.
    await context.add_requests(
        [
            Request.from_url(
                context.request.url,
                session_id=context.session.id,  # Pins the retry to this session.
                user_data={'prepared': True},
                always_enqueue=True,  # Bypasses deduplication so the same URL runs again.
            )
        ]
    )

Option 3 — re-apply cookies in a pre-navigation hook

Keeps the raise-to-retry flow and rotation, at the cost of some manual plumbing. Snapshot the cookies after the setup step into an external store keyed by the request's unique_key, then re-apply them onto whichever session the retry picks in a pre_navigation_hook (which runs before navigation and receives context.session).

from crawlee.crawlers import HttpCrawler, HttpCrawlingContext

crawler = HttpCrawler()
pending_cookies: dict[str, list] = {}


@crawler.router.default_handler
async def handler(context: HttpCrawlingContext) -> None:
    if not has_what_i_need(context):  # your existing check, e.g. cookies present
        # First pass: establish cookies, then snapshot them for the retry.
        await context.send_request('https://example.com/set-something')
        pending_cookies[context.request.unique_key] = context.session.cookies.get_cookies_as_dicts()
        raise RuntimeError('retry with cookies')

    pending_cookies.pop(context.request.unique_key, None)
    # ... do the real work; cookies are on context.session ...


@crawler.pre_navigation_hook
async def restore_cookies(context: HttpCrawlingContext) -> None:
    if saved := pending_cookies.get(context.request.unique_key):
        context.session.cookies.set_cookies(saved)

Note: use an external store (as above), not context.request.user_data, to carry the snapshot across the retry. During a handler run context.request is an isolated copy, so in-handler mutations to it (including user_data) don't survive the retry (see #2061).

The guide should also note that cookies are scoped by domain (the send_request target must set them for the same host you're crawling), and that a custom HTTP client needs to attach the session's cookies to outgoing send_request calls.

Notes for a future redesign (v2)

Reusing a session across retries isn't ergonomic today, and we expect to rethink this in Crawlee for Python v2 / Crawlee for JS v4. Worth capturing here as prior art / parity references:

• Crawlee for JS Session management and its SessionPool / sessionPoolOptions.maxPoolSize.
• Related parity gap: in Crawlee for JS, request.userData mutations persist across retries (same Request instance), whereas in Crawlee for Python context.request is an isolated deepcopy during handler execution, so in-handler mutations to context.request (including user_data) are silently dropped on retry. That difference makes some of the natural workarounds (stash state on request.user_data, then read it on the retry) fail. This is worth deciding on explicitly as part of the redesign.

[vdusek]

@Mantisus Could you take a look? This is a question I've seen many times, so I think we should address it in the docs.

[Mantisus]

Good point. I'll add a guide on working with cookies.