How to Use Residential Proxies with Scrapy in 2026

Key Takeaways

• Connect residential proxies to Scrapy using Request.meta["proxy"] or project-wide proxy environment variables.
• Choose the right proxy setup based on your needs: a rotating residential gateway, proxy pool, or sticky session.
• Use rotating residential proxies to distribute requests across different residential IPs without managing IP rotation yourself.
• Keep proxy credentials secure by using environment variables, deployment secrets, or a secrets manager instead of hardcoding them.
• Verify your proxy setup before scaling by checking the exit IP, location, response status, and IP behavior across multiple requests.

If you use Scrapy to crawl websites at scale, sending every request directly from your own IP can lead to rate limits, blocked requests, or location-based restrictions.

Residential proxies let you route Scrapy traffic through residential IP addresses instead. The good news is that you don't need a complicated proxy-rotation system to get started. Scrapy's built-in HttpProxyMiddleware supports proxies through the Request.meta["proxy"] value, including authenticated proxy URLs.

In this guide, you'll learn how to connect a residential proxy to Scrapy, route every request through it, rotate residential IPs, use sticky sessions and geo-targeting, troubleshoot common errors, and optimize your crawler.

How Do Residential Proxies Work with Scrapy?

A residential proxy acts as an intermediary between your Scrapy crawler and the website you're scraping. Instead of sending requests directly from your server's IP address, Scrapy sends them to a proxy endpoint, which forwards the requests through a residential IP. The target website therefore sees the residential IP rather than your crawler's original IP.

The basic flow is:

There are two common ways to route Scrapy traffic through residential proxies:

1. Use a provider-managed residential proxy gateway: Scrapy connects to one proxy endpoint, while the provider handles IP selection and rotation.
2. Manage a proxy pool in Scrapy: Your application maintains multiple proxy endpoints and decides which one to use for each request.

If you use a provider-managed gateway, you can typically choose between rotating IPs and sticky sessions depending on whether you want the IP to change or remain consistent across multiple requests.

The right approach depends on whether you want the proxy provider to manage IP rotation or need more control over individual requests. 

Use a rotating residential proxy gateway

With a rotating residential proxy, Scrapy can keep sending requests to the same proxy endpoint while the proxy service handles IP selection and rotation in the background. For example:

Request 1

Request 2

From Scrapy's perspective, both requests go to the same proxy endpoint, but the proxy service determines which residential IP handles each request. This means you don't need to maintain a list of IPs or build your own rotation logic 

Use a proxy pool 

With a proxy pool, your Scrapy project has multiple proxy endpoints and decides which one to use for each request. 

Request 1

Request 2

This gives you more control over proxy selection, but your application is now responsible for rotation, removing failed proxies, and retrying requests through another proxy.

Which approach should you use?

• Rotating residential gateway: Best for most Scrapy projects. The proxy provider handles IP selection and rotation, so you don't need to build that logic into your crawler.
• Proxy pool: Best when you need control over which proxy handles each request, along with your own rotation and proxy management logic.
• Sticky session: Best when multiple requests need to come from the same residential IP, such as maintaining a session or completing a multi-step workflow. 

For most users, start with a managed rotating residential endpoint. Build custom middleware only when the requirements justify it.

How to Set up A Residential Proxy in Scrapy

Once you have your residential proxy credentials, connecting it to Scrapy only takes a few steps. You'll need your proxy host, port, username, and password, then you can pass the proxy endpoint to your Scrapy requests. Before adding it to your spider, it's worth testing the proxy separately so you can confirm the connection works before troubleshooting your Scrapy setup. 

Step 1: Get your residential proxy credentials

Before changing your Scrapy project, get the connection details from your proxy provider. You'll typically need:

• Proxy host
• Proxy port
• Username
• Password
• Proxy protocol, such as HTTP or SOCKS5
• Optional country, city, ISP, or session settings

For example, your credentials might look conceptually like this:

Some providers also let you configure the location and IP behavior associated with an endpoint.

Using Proxidize Residential Proxies with Scrapy

If you're using Proxidize, you can get the proxy connection details directly from the Residential Proxies dashboard. After logging in to your Proxidize account:

1. Go to Residential Proxies and open the residential proxy dashboard.
2. Create or configure an access point based on your requirements. You can select options such as country, city, ISP, stickiness, and hostname.
3. The dashboard then provides the connection details for your endpoint, including the proxy host, port, username, and password.
4. Use these values in the same Scrapy meta["proxy"] configuration shown above.

For example:

You can modify the endpoint settings from the Residential Proxies dashboard whenever your crawler needs a different location or IP behavior. This lets you configure the proxy first and then use the resulting endpoint directly in your Scrapy project.

Step 2: Test the proxy before adding it to Scrapy

Test the proxy independently before debugging your Scrapy project. For example, if your provider gives you an HTTP proxy, you can test the connection with cURL:

If the request succeeds, the returned IP should be the IP assigned through the proxy rather than your normal public IP. This simple test separates proxy problems from Scrapy problems. If the proxy doesn't work independently, changing your Scrapy code won't fix the underlying connection.

Step 3: Add the residential proxy to a Scrapy request

Once you've confirmed that the proxy works, add it to your Scrapy request using meta["proxy"]:

That's enough for a basic proxy connection. Scrapy's built-in HttpProxyMiddleware reads the proxy request metadata and supports proxy URLs in the form http://host:port or http://username:password@host:port. The per-request proxy value takes precedence over the http_proxy and https_proxy environment variables.

What does the proxy URL mean?

In this example:

each part has a specific purpose:

• http:// - the proxy protocol
• USERNAME - your proxy username
• PASSWORD - your proxy password
• HOST - the proxy server hostname
• PORT - the proxy server port

Keep your proxy credentials secure

Avoid hardcoding your residential proxy username and password directly in your Scrapy code, especially if the project is stored in a shared repository or deployed to production. Instead, store your proxy credentials using:

• Environment variables for local development
• Deployment secrets for hosted environments
• A secrets manager for production applications

For example:

You can then use proxy_url when assigning the proxy to your Scrapy requests.

If your password contains special characters such as @, :, /, or #, URL-encode the credentials before adding them to the proxy URL.

Step 4: Run the spider and verify the proxy is being used

Don't assume the proxy is working just because the spider runs without an obvious error. Verify:

1. The request returns a valid response
2. The exit IP is different from your normal IP
3. The IP belongs to the expected location
4. The target website returns the expected content

For example, temporarily point your spider at an IP-checking endpoint:

You can then inspect the response and confirm which IP your request used.

How to use Proxidize with Scrapy? 

Once you've confirmed that your Scrapy proxy configuration works, you can use the same setup with Proxidize residential proxies. Configure your Proxidize proxy based on the location and IP behavior your crawler requires, then pass the Proxidize endpoint to Scrapy using meta["proxy"].

For example, a Proxidize endpoint can be configured in Scrapy like this:

Depending on your crawler's requirements, Proxidize supports rotating and sticky residential sessions, geographic targeting, and HTTP(S)/SOCKS5 connections. This lets you choose the proxy configuration based on whether your Scrapy project needs frequent IP rotation, a consistent IP across a session, or traffic from a specific location.

How To Use a Residential Proxy for Every Scrapy Request

If you only need a proxy for specific requests, adding meta["proxy"] to each request is straightforward, but when your entire Scrapy spider needs to use the same residential proxy, adding it manually to every request can quickly become repetitive.

There are two ways to route every Scrapy request through a residential proxy:

1. Environment variables: The simplest option when you want the same proxy configuration across your entire Scrapy project.
2. Custom downloader middleware: Better when you need more control over proxy selection, routing, rotation, retries, or proxy health.

Set the proxy through environment variables

Scrapy's built-in HttpProxyMiddleware supports the standard http_proxy, https_proxy, and no_proxy environment variables. For example:

Then run your spider as usual:

This routes your project's HTTP and HTTPS requests through the configured proxy without adding meta["proxy"] to every request.

Global proxy vs. per-request proxy

Scrapy doesn't have built-in PROXY_HOST or PROXY_PORT settings in settings.py that automatically apply a proxy to every request. Its built-in proxy middleware uses the proxy request metadata or standard proxy environment variables.

If you need Scrapy to choose different proxies based on the request, rotate through a proxy pool, or handle proxy failures automatically, custom downloader middleware is the better option.

How to Rotate Residential Proxies in Scrapy

If you're sending a large number of requests through Scrapy, rotating residential IPs can help distribute traffic across different IP addresses. There are two main ways to do this:

1. Use a rotating residential proxy endpoint that handles IP rotation for you
2. Manage a pool of proxies yourself with Scrapy middleware. 

The simpler option depends on how much control you need over the rotation process. 

Option 1: Use a rotating residential proxy endpoint

This is usually the simplest approach. Instead of maintaining a list of IP addresses yourself, Scrapy connects to one proxy endpoint:

Your Scrapy code can remain as simple as:

The provider handles the proxy-side rotation, you don't need to build and maintain a proxy pool inside Scrapy.

Option 2: Rotate proxies with Scrapy middleware

Custom middleware makes sense when you control a pool of proxy endpoints and need Scrapy to choose between them. For example:

Then configure the middleware in settings.py:

Scrapy downloader middleware is specifically designed to modify requests and responses globally, and middleware order determines when each component runs in the request/response chain.

A simple random selector is enough for demonstrating the concept, but production proxy rotation usually needs more than random.choice(). You may also want to track failed proxies, retry failed requests through another proxy, manage sessions, and avoid repeatedly selecting unhealthy endpoints.

How to use sticky residential sessions with Scrapy

If you're scraping a website that requires several requests to complete the same workflow, switching IPs between those requests can make the requests appear to come from different clients.

For example:

A sticky residential session keeps the same residential IP across those requests.

Proxidize supports both location targeting and session-based proxy configurations, so you can choose residential IPs based on your required location and maintain a sticky session when your workflow needs the same IP across multiple requests. For Scrapy projects that need frequent IP changes, a rotating setup can handle IP rotation without requiring you to manage the proxy pool yourself. 

How to Handle Proxy Authentication in Scrapy

Most residential proxy services require authentication before allowing your requests through the proxy. Depending on the provider, you may authenticate with a username and password or through IP allowlisting.

Use username and password authentication

The simplest method is to include your credentials in the proxy URL:

meta={

   "proxy": "http://USERNAME:PASSWORD@HOST:PORT"

}

Scrapy's HttpProxyMiddleware supports authenticated proxy URLs in this format. If your provider supports IP allowlisting, you may not need to include credentials in the proxy URL. 

What if Scrapy returns 407 Proxy Authentication Required?

A 407 means the proxy requires authentication but doesn't accept the credentials provided. Check your:

• Username and password
• Proxy host and port
• Proxy URL format
• Protocol
• URL encoding for special characters in credentials

If you're still unsure whether the problem is with Scrapy or the proxy, test the same credentials with cURL first.

Common Residential Proxy Errors in Scrapy

Once your residential proxy is connected, most issues come down to the proxy credentials, endpoint, or connection itself. The error message usually points you toward the problem, so checking the status code and proxy configuration first can save you a lot of debugging time. 

The fastest troubleshooting approach is to isolate the problem:

How to Monitor Scrapy Proxy Performance at Scale? 

Once your residential proxy is working correctly, the next step is making sure it continues to perform reliably as your Scrapy crawler handles more requests. Monitor key performance metrics such as success rates, 403 and 429 responses, latency, retries, bandwidth usage, and IP behavior to identify issues before they affect your crawl. 

• Success rate: Percentage of requests returning the expected response.
• 403 and 429 rates: Track whether targets are increasingly rejecting or rate-limiting requests.
• Latency: Monitor response times to identify slow proxies or overloaded routes.
• Retry rates: A rising retry rate can indicate proxy health or configuration issues.
• Bandwidth usage: Especially important for residential proxies because usage is typically bandwidth-based.
• IP behavior: Monitor whether IPs are rotating as expected or whether the same IP is appearing too frequently.
• Response codes / errors: Track 5xx errors, connection failures, timeouts, etc.

Track these metrics over time rather than judging the proxy from a handful of requests. If 403s, 429s, retries, latency, or unexpected IP behavior increase as volume grows, adjust your rotation, concurrency, session settings, or proxy configuration. 

What's Next for Your Scrapy Setup? 

Getting residential proxies working with Scrapy is only the first step. Once you've confirmed that your requests are reaching the target through the expected residential IP, the next step is to build a setup that can handle your actual crawling workload.

Start with a rotating residential endpoint if you want the proxy provider to manage IP rotation for you. As your crawler grows, you can decide whether you need sticky sessions for stateful workflows or custom middleware for more control over proxy selection and routing.

From there, focus on how your crawler behaves at scale:

• How many requests you're sending
• How much proxy bandwidth you're using, and 
• Whether your current rotation strategy fits the sites you're scraping. 

Keep your setup as simple as possible, and only add more proxy logic when your crawler actually needs it. Once your Scrapy crawler is working with residential proxies, you have the foundation to scale your scraping projects without managing every residential IP yourself.

If you’re ready to put your Scrapy setup into practice, you can try it with Proxidize residential proxies. Proxidize provides rotating residential IPs, flexible targeting, and session options, so you can connect your proxy endpoint to Scrapy without managing a proxy pool yourself. 

Try Proxidize Today