docs: Add Migrating from Scrapy guide

[Mantisus]

Description

• Add a new guide for migrating from Scrapy to Crawlee. It provides a mapping of the core concepts and a set of side-by-side examples (Scrapy vs Crawlee) covering common cases, from a basic spider migration to error handling, login, and JavaScript rendering.

Issues

• Closes: Write a migration guide from Scrapy to Crawlee #1461

[Copilot]

Pull request overview

Adds a dedicated “Migrating from Scrapy” documentation guide to help Scrapy users translate core concepts and common patterns (routing, data passing, throttling, proxies, error handling, login, and JS rendering) into Crawlee for Python.

Changes:

• Introduces a new docs/guides/scrapy_migration.mdx guide with a concept mapping table and side-by-side Scrapy vs Crawlee examples.
• Adds Scrapy and Crawlee code snippets under docs/guides/code_examples/scrapy_migration/ to support the guide’s runnable and comparison blocks.
• Updates Ruff per-file ignores for the Scrapy snippet files used purely for comparison.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyproject.toml	Adds Ruff per-file ignores for Scrapy tutorial snippets included as docs-only comparison code.
docs/guides/scrapy_migration.mdx	New migration guide content with concept mapping and side-by-side examples.
docs/guides/code_examples/scrapy_migration/scrapy_throttling.py	Scrapy AutoThrottle settings snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_quotes.py	Scrapy quotes spider snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_proxy.py	Scrapy rotating proxies/settings snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_playwright.py	Scrapy + scrapy-playwright spider snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_playwright_settings.py	scrapy-playwright settings snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_formrequest.py	Scrapy FormRequest login snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_export.py	Scrapy FEEDS export settings snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_errback.py	Scrapy errback/retry behavior snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_crawlspider.py	Scrapy CrawlSpider/Rules snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_concurrency.py	Scrapy concurrency/download delay settings snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_cb_kwargs.py	Scrapy cb_kwargs snippet for comparison.
docs/guides/code_examples/scrapy_migration/scrapy_authors.py	Scrapy callback routing snippet for comparison.
docs/guides/code_examples/scrapy_migration/crawlee_user_data.py	Crawlee equivalent showing Request.user_data + labeled handlers.
docs/guides/code_examples/scrapy_migration/crawlee_throttling.py	Crawlee equivalent showing ThrottlingRequestManager usage.
docs/guides/code_examples/scrapy_migration/crawlee_quotes.py	Crawlee equivalent of the canonical quotes spider.
docs/guides/code_examples/scrapy_migration/crawlee_proxy.py	Crawlee equivalent proxy rotation via ProxyConfiguration.
docs/guides/code_examples/scrapy_migration/crawlee_post.py	Crawlee equivalent of form submission/login with POST payload + headers.
docs/guides/code_examples/scrapy_migration/crawlee_playwright.py	Crawlee PlaywrightCrawler example for JS-rendered pages.
docs/guides/code_examples/scrapy_migration/crawlee_labels.py	Crawlee equivalent of callback routing using request labels and router handlers.
docs/guides/code_examples/scrapy_migration/crawlee_export.py	Crawlee equivalent dataset export via export_data().
docs/guides/code_examples/scrapy_migration/crawlee_error_handling.py	Crawlee equivalent error + failed-request handlers aligned with retry lifecycle.
docs/guides/code_examples/scrapy_migration/crawlee_crawlspider.py	Crawlee equivalent rule-based crawling via enqueue_links() with include globs and labels.
docs/guides/code_examples/scrapy_migration/crawlee_concurrency.py	Crawlee equivalent concurrency/rate settings via ConcurrencySettings.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vdusek]

Hi @Mantisus, I didn't tell you, request review from @szaganek only for the docs changes in SDK & Client repos.

[Mantisus]

Hi, @vdusek. Oh, got it, thanks for the information. @szaganek, sorry for the unnecessary ping.

[vdusek]

Just a few comments, but overall, this looks great 👍

[vdusek]

The Scrapy counterparts don't set the max requests arg, is that intentional?

[Mantisus]

Yes. This is mentioned at the beginning before ## Installation, otherwise, we need to add settings.py to every example just for this setting.

[vdusek]

A few more from Claude

[Mantisus]

@honzajavorek, would you mind taking a look at this guide? I'd appreciate your feedback.