Guide to Using Scrapy Playwright

Scrapy Playwright connects Scrapy's crawling and data-export features with Playwright's browser automation. Use it when the information you need appears after JavaScript runs, or when collecting it requires an action such as scrolling. This guide walks through installation, a complete spider, JSON export, and optional Proxidize proxy configuration. The example visits a public practice site, waits for quotes to appear, scrolls once, verifies that the rendered quote count increases, and saves the results.

Quick Answer

Scrapy Playwright lets selected Scrapy requests run through a Playwright browser, allowing a spider to wait for JavaScript-rendered elements, perform actions such as scrolling, and parse the resulting HTML with Scrapy. Use it only when a page needs browser rendering or interaction. When routing, location, or session control is required, configure the proxy at the Playwright browser-context level.

Key Takeaways

• Add "playwright": True only to requests that need browser rendering; other requests continue through Scrapy's ordinary downloader.
• This walkthrough uses async def start(), which requires Scrapy 2.13 or newer, and installs the Playwright browser separately from the Python packages.
• Wait for required elements and verify that an action changed the page instead of relying only on fixed delays.
• A Playwright context proxy covers browser requests in that context, not ordinary Scrapy downloader traffic such as the separate robots.txt request in this example.
• Keep one proxy session for dependent browser actions, protect credentials outside the source code, and validate the exported records rather than treating process completion as proof of a successful crawl.

What Is Scrapy Playwright?

Scrapy Playwright is an integration between Scrapy and Playwright. It allows users to scrape dynamic pages with Scrapy by processing selected web scraping requests through a Playwright browser. It also exposes Playwright features such as mouse and keyboard actions, event and element waits, screenshots, and custom JavaScript.

Playwright is an open-source browser automation library released by Microsoft in 2020. It supports Chromium, Firefox, and WebKit and is used for browser testing, automation, and data collection. Read the headless browser guide for the underlying browser model or compare it with Puppeteer alternatives.

Scrapy is a Python crawling framework with request scheduling, item pipelines, exports, middleware, and other components for structured collection. The Scrapy web-scraping guide explains the framework without the browser integration. Scrapy's ordinary downloader is usually preferable when the required data is already present in the initial response; use Playwright only for requests that need browser rendering or interaction.

Some teams use Scrapy Splash for JavaScript rendering. This tutorial uses Scrapy Playwright because its current API integrates Playwright actions with Scrapy requests and responses.

Install Scrapy Playwright

Use Python 3.12 or newer for this walkthrough. The code uses async def start(), which requires Scrapy 2.13 or newer. Check your installed Python version before creating a virtual environment.

On macOS or Linux:

On Windows PowerShell:

With the environment activated, install the Python packages and the Chromium browser:

Installing the Python packages and installing the browser are separate steps. On a supported Linux machine that reports missing browser system libraries, use Playwright's dependency installer:

Create the Scrapy project and enter the directory containing scrapy.cfg:

The official Scrapy Playwright documentation lists the integration's current minimum dependencies. Playwright's browser installation guide covers browser binaries and supported system dependencies.

Configure Scrapy to Use Playwright

Open playwright_scraper/settings.py. Add or update these settings, replacing duplicate assignments if they already exist:

Requests use the browser only when their metadata contains "playwright": True. Other requests continue through Scrapy's ordinary downloader. Setting USER_AGENT to None lets Playwright use the browser's normal user agent. ROBOTSTXT_USER_AGENT gives Scrapy's robots.txt middleware a separate non-empty identity; current Scrapy versions require one when robots rules are enabled.

This example disables automatic retries to make a failed run easy to diagnose. Add bounded retries only after you understand the failure.

Create a Spider That Waits and Scrolls

Create playwright_scraper/spiders/quotes_browser.py and paste the complete file below.

The target is Quotes to Scrape's infinite-scroll exercise. The example waits for the initial quotes, scrolls once, and verifies that the rendered count increases. The practice page may load more than one API batch while it settles or while Playwright captures the full-page screenshot, so the spider does not assume a fixed final count or claim to collect every quote on the site.

The browser waits for actual quote elements, scrolls, and waits until the number of quotes increases. If that does not happen within 15 seconds, the request fails instead of presenting an unchanged first view as a successful scrolling example.

PageMethod runs the browser actions before Scrapy parses the resulting HTML. The function returns the count measured before scrolling, while parse() measures the rendered response that Scrapy will actually export. This avoids reporting a stale count if the practice page loads another batch before the browser response is serialized.

The spider leaves playwright_include_page unset, so the integration closes each page automatically. If you change the example to pass a page into your own callbacks, close that page on both success and failure.

Run the Spider and Inspect the Output

Run this command from the folder containing scrapy.cfg:

-O overwrites the output file for each run. A successful run produces:

• quotes.json, containing records with text, author, tags, and source_url.
• quotes.png, showing the page after scrolling.
• A log entry showing that the quote count increased after scrolling.

Check the exported file:

One record from the final verified proxy run had this shape:

Add a Proxidize Proxy

First complete a successful direct run. Then copy the HTTP proxy server, username, and password generated for your Proxidize access point. Use the server's actual scheme, hostname, and port; do not guess them or paste credentials into the article or source control.

Append this block to playwright_scraper/settings.py:

To enter credentials without embedding them in the source file or command history, create run_with_proxy.py beside scrapy.cfg:

Run it from a terminal:

Inspect quotes-proxy.json and the crawler log using the same checks as the direct run. Scrapy can finish with a process exit code of zero even when an individual request failed, so check the exported item count and errors as well.

This configuration applies the proxy to the default Playwright browser context. It does not proxy ordinary Scrapy requests, such as a robots.txt fetch through the ordinary downloader. Configure those separately if the entire crawler must use a proxy.

Keep a stable proxy session while a page loads its related assets and scrolls through dependent results. A browser context and a proxy session are different things: use Proxidize's generated configuration to select the appropriate session behavior. When the proxy address or credentials change, restart this example so it creates a new context. Reusing an existing context does not apply new context options.

For protocol differences, read HTTP vs. SOCKS5 proxies. Choose Residential Proxies for broad geographic coverage or Mobile Proxies when real mobile-carrier routing is an intentional requirement. The products provide the network route; the crawler still owns browser behavior, validation, pacing, and compliance with applicable rules.

Troubleshoot Common Failures

Tested Environment and Results

This walkthrough was tested end to end on September 23, 2026, on Linux 6.11 x86-64 with:

• Python 3.12.14
• Scrapy 2.19.0
• scrapy-playwright 0.0.48
• Playwright 1.63.0
• Chromium for Testing 153.0.8010.12, Playwright build 1243

The verified direct run started with 10 quotes and exported 40 valid records after scrolling and full-page capture. The final verified Proxidize run, executed through the interactive run_with_proxy.py helper, started with 10 and exported 20 valid records. Another authenticated run loaded additional batches, so the exact count is not a fixed expectation. The practice page may load one or more API batches while its scroll handler and full-page screenshot settle; success means the count increased, the logged final count matched the export, every record contained text and an author, and the crawl had no request or spider errors.

A separate browser-based IP check returned an exit different from the test host's direct public IP. In the deliberately incorrect-password test, navigation timed out and the export contained zero records. No successful direct fallback was observed during this test. A timeout alone does not prove how every possible proxy failure will behave, so production code should still validate the observed route and returned data. No proxy credentials or complete diagnostic IP address are included here.

Conclusion

Scrapy Playwright lets Scrapy collect content that appears after JavaScript runs or after a browser action. A reliable example needs more than a browser launch: wait for the required elements, verify the requested action completed, validate exported fields, and handle failures. Add a proxy when the task requires its routing or location controls. Proxies do not guarantee access to a website or replace correct browser and session handling.