How to Use Residential Proxies With CrewAI Agents

Quick Answer

To use residential proxies with CrewAI, put the proxy configuration on the custom HTTP or browser tool that opens target websites. CrewAI coordinates agents, tasks, crews, flows, model calls, and tools; it does not automatically route a tool's network traffic.

For a multi-agent Crew, give each network-facing agent its own Proxidize Residential Proxy work-unit session:

Recommended production pattern: The live validation for this guide used one US-scoped Random residential route. UK, Germany, three independent credentials, and Sticky continuity were not live-tested.

The residential endpoints carry website data-plane traffic. Proxidize MCP is a separate control-plane connection. A successful MCP call does not prove that a page request used the residential proxy, and a proxied page request does not mean the model API or MCP connection used that proxy.

The essential HTTPX configuration is:

Wrap the client in a CrewAI BaseTool, then give that tool only to the agent assigned to that route. For coherent multi-page research, use one Sticky session for one work unit and keep its HTTP connection pool, cookie jar, evidence, and concurrency budget isolated. Use rotating or Random behavior for unrelated one-shot samples where continuity is unnecessary.

Responsible-use note: Use agents and proxies only for lawful, authorized work. Follow applicable laws, privacy requirements, website terms, access controls, and reasonable request rates. A proxy changes the network route; it does not grant permission to collect data or guarantee that a website will accept a request.

Methodology: CrewAI behavior and package versions were checked against current first-party CrewAI documentation and PyPI artifacts on September 2, 2026. Proxidize capabilities were checked against the current Residential Proxy and MCP pages. The exact example passed compilation, construction, schema, three-agent concurrency, authenticated HTTP CONNECT, HTTPX, Playwright, URL-policy, evidence-isolation, citation-gate, and native CrewAI MCP tests with controlled fixtures. The credentialed US-only run then used crewai 1.15.18, openai/gpt-5.6-terra through CrewAI's Responses API mode, HTTPX 0.28.1, Playwright 1.62.0, one US-scoped Random Proxidize Residential credential, and @proxidize/mcp 0.1.3. Two independent geo services classified the exit as US; live HTTPX, Playwright, the real model/tool/manager loop, direct MCP discovery, a read-only MCP operation, and CrewAI's one-tool native MCP allowlist all passed. The run did not test UK or Germany, three independent proxy credentials, target-visible localization, Sticky configuration or duration, production concurrency, or target-specific behavior.

Key Takeaways

• Proxy the network-aware tool, not CrewAI as a whole. Configure Proxidize on HTTPX, Playwright, Selenium, or the specific client that opens the website.
• Give concurrent agents isolated work-unit sessions. Each specialist should have its own generated credentials, client or browser context, cookies, request budget, and evidence namespace.
• Use Sticky for coherent browsing. A research or verification task that spans multiple pages usually benefits from a consistent exit. Rotate between independent jobs, tenants, or market observations.
• A Random endpoint can still fetch pages, but it does not provide session continuity. Do not call a route Sticky unless the access point was configured that way and continuity was observed.
• Configured geography is not verified geography. Check the target's visible currency, store, language, availability, or other market-specific fields. Locale and timezone settings are supporting signals, not proof.
• CrewAI concurrency and network concurrency are separate. async_execution=True can overlap specialist tasks, while an application semaphore must still bound real HTTP or browser work.
• Crew and Flow state are not proxy or browser state. Restoring task progress does not restore an exit lease, TCP connection, cookie jar, browser process, or target-side session.
• Treat page content as untrusted. Enforce destination policy, redirect checks, size limits, tool budgets, and citation validation in code rather than trusting instructions in prompts.
• MCP is a control plane. Use Proxidize MCP for narrowly approved settings, locations, usage, or access-point operations; keep website traffic on the residential endpoint.

What Is CrewAI?

CrewAI is a framework for building collaborative AI agents and event-driven automations. Its core abstractions separate responsibilities:

• An Agent has a role, goal, model, and permitted tools.
• A Task describes work, expected output, assigned agent, and optional dependencies.
• A Crew coordinates agents and tasks through a sequential or hierarchical process.
• A Flow provides event-driven control, explicit application state, branching, and orchestration around crews or other work.

That separation makes CrewAI a natural place to enforce proxy boundaries. The Crew decides which specialist performs a task. The specialist receives one narrowly scoped web tool. The tool owns the HTTP client or browser and its Proxidize credentials.

The current stable Python packages used for this guide are crewai, pinned as crewai==1.15.18, and crewai-tools, pinned as crewai-tools==1.15.18; both were released August 27, 2026. Newer date-stamped development builds may appear on PyPI, but this guide pins the latest stable release rather than a prerelease.

CrewAI's current documentation has a dedicated Web Scraping & Browsing tool category. It includes first-party documentation for direct scraping, Selenium, hosted browsers, Stagehand, Firecrawl, and multiple proxy or data providers. You are not required to use a packaged web tool, however. A custom BaseTool is often the clearest option when you need exact proxy assignment, destination controls, response limits, and evidence tracking.

What CrewAI does—and what the proxy tool does

What Is Actually Being Proxied?

Only clients explicitly configured with a Proxidize endpoint send their requests through the residential route.

This gives you three different proofs to collect during testing:

1. The model and Crew can complete a tool loop.
2. The website tool actually traverses the intended proxy route.
3. The target returns the intended market-specific experience.

None of those automatically proves the other two.

Avoid setting HTTP_PROXY, HTTPS_PROXY, or ALL_PROXY for the entire agent process unless routing every compatible client is intentional. Global variables can accidentally proxy model APIs, telemetry, storage, callbacks, package downloads, and MCP traffic. This example uses trust_env=False on HTTPX and passes proxy settings directly to the page client.

Why Pair CrewAI With Residential Proxies?

A CrewAI application may need lawful public information as it appears from several residential markets. Common examples include:

• Comparing public prices, currency, inventory, or shipping options across regions.
• Monitoring localized search results, landing pages, storefronts, or marketplace listings.
• Collecting fresh, geographically diverse evidence for market research, RAG, or evaluation.
• Verifying a specialist's result from a separate market or network session.
• Reviewing public listings for authorized brand-protection work.
• Testing an approved website's visible localization behavior.

Proxidize Residential Proxies currently provide millions of ethically sourced residential IPs across 195+ countries, country/city/ISP targeting, rotating and sticky sessions, HTTP(S) and SOCKS5 support, and dashboard/API control. The integration uses standard proxy settings, so the CrewAI tool does not need a proprietary network SDK.

Residential proxies are generally the better product fit for global, geo-diverse web research. Mobile proxies are a narrower choice when the work specifically requires a real US mobile-carrier route, mobile identity, or carrier targeting.

A residential proxy is not necessary for every Crew. If the agents work with internal systems, licensed datasets, local documents, your own APIs, or an approved first-party feed, a public-web proxy layer can add cost and operational complexity without improving the result.

Want to test the architecture? Create a Proxidize Residential Proxy access point for each independent work unit, then place the generated server and credentials in the corresponding environment variables below. For the broader product fit, see AI Agent Proxies for Reliable Web Access.

Design One Proxy Session per CrewAI Work Unit

The useful CrewAI-specific pattern is not merely “several agents use a proxy.” It is “each network-facing specialist owns an isolated network work unit.”

Isolation should include more than a route label:

• Separate generated proxy credentials or session identifiers.
• A separate HTTP connection pool or browser context.
• A separate cookie jar and target-side session.
• A separate tool-call budget.
• A per-route lock if one coherent session must not overlap its own requests.
• A route-specific allowlist.
• A route-specific evidence namespace.
• A clean lifecycle and explicit teardown.

The proxy hostname and port may be identical across access points. The username, access-point configuration, or session selector can still determine the location and Sticky assignment. Keep those generated values opaque. Do not let the model create suffixes, rewrite usernames, or choose arbitrary session IDs.

Why one shared client is not enough

If three specialists share one HTTP client or browser context, they may also share:

• Cookies and authentication state.
• TCP connection reuse.
• Cache entries.
• Browser storage.
• A Sticky exit assignment.
• Rate-limit state.
• Evidence and logs.

That can contaminate market comparisons and cross tenant boundaries. A UK price result fetched after a US navigation may reflect the wrong cookies even when the proxy exit changes. Conversely, a single Sticky proxy with three cookie jars can still make all specialists appear from one network location.

The article's example creates three RouteSession objects and three ProxiedPageTool instances. The manager receives no target-website tool.

Prerequisites

For the exact package set in this guide, you need:

1. Python 3.10–3.13. The controlled validation used Python 3.11.
2. crewai==1.15.18.
3. A model supported by CrewAI and its corresponding provider credential.
4. An active Proxidize Residential Proxy plan.
5. Three separately generated work-unit routes if you want simultaneous US, UK, and Germany observations.
6. Exact hostnames and HTTPS start URLs for websites you are authorized to access.
7. Chromium for the optional Playwright path.
8. Node.js 18 or later and a fresh Proxidize API token only if you enable @proxidize/mcp.

Create a virtual environment and install the pinned packages:

The custom BaseTool and native MCP configuration come from crewai; crewai-tools is included because many real CrewAI projects use its browser/scraping integrations and advanced MCPServerAdapter. You can remove it if your locked application does not import it.

Configure secrets and route policy

Use a secret manager in production. A local .env file can be convenient for development, but keep it out of version control:

The IANA URLs are neutral integration targets; they do not display US, UK, or Germany-specific commerce fields. Leave *_EXPECTED_MARKERS empty for that smoke test and make no geography claim. For a real authorized target, set deterministic markers for the fields that matter—such as an expected currency label, store name, or shipping region—after reviewing the target's format. A marker match is evidence about page content, not proof of an IP's physical location.

The model example uses openai/gpt-5.6-terra, a current function-calling model documented by both CrewAI and OpenAI. It selects CrewAI's Responses API mode and uses max_completion_tokens plus low reasoning effort instead of assuming every model accepts the same sampling controls. The proxy boundary does not depend on OpenAI: replace CREWAI_MODEL and set the corresponding provider credential if your Crew uses another supported model.

Build a Custom HTTPX Tool

HTTPX is the best first transport when the required information is already present in an HTML, JSON, or text response. It is easier to bound and usually consumes less residential bandwidth than a full browser.

1. Keep proxy credentials separate

Do not build one credential-bearing string and then log it. Store the server, username, and password separately and combine them only inside the client:

Then create a route-owned client:

trust_env=False prevents unrelated process-level proxy variables from silently changing this client's route. follow_redirects=False makes every redirect an application decision.

2. Validate every destination and redirect

An agent-controlled URL is an SSRF boundary. The example accepts only:

• HTTPS URLs.
• Port 443.
• Exact, IDNA-normalized hostnames on a route-specific allowlist.
• URLs with no embedded username or password.
• Hostnames whose local resolution contains only globally routable addresses.

The same validation runs after every redirect. A starting URL on an approved host does not make a redirect to a private IP, metadata service, or unapproved host safe.

Exact hostname matching is deliberate. Allowing every subdomain through a suffix check can expose abandoned or user-controlled subdomains. Add each required hostname after review.

3. Bound response work

The tool also limits:

• Redirect count.
• Retry attempts.
• Request duration.
• Response media types.
• Downloaded bytes.
• Extracted text length.
• HTTP connections per route.
• Concurrent work across the process.
• Tool calls per agent.

Raw web pages are noisy and may contain indirect prompt injection. Return the smallest task-relevant evidence you can. For a price workflow, a production tool should ideally return fields such as product ID, price, currency, availability, market label, final URL, and observation time—not 12,000 characters of arbitrary page text.

4. Expose one route as one CrewAI tool

CrewAI custom tools subclass BaseTool and define an input schema plus _run:

The model-visible schema contains only url. The route, username, password, cookie jar, client, semaphore, and evidence registry stay in application-owned private state.

The example uses a module-level never_cache callback because inline lambdas cannot be serialized for CrewAI checkpointing. It also disables Crew-level caching: a market observation should not silently reuse evidence collected through another session or at another time.

Build the Multi-Agent Crew

CrewAI can overlap consecutive tasks with async_execution=True. A later synchronous task with those tasks in context waits for them to complete. That maps directly to three independent market specialists followed by one manager.

Create the Crew with a sequential process:

“Sequential” describes task ordering semantics; it does not mean every task is forced to run one at a time. CrewAI starts the three consecutive async tasks, waits for them before the synchronous manager task, and supplies their output as explicit context.

The manager has no HTTPX or Playwright tool. It cannot open a fourth page, switch markets, or silently replace specialist evidence. It can only synthesize the task context. When optional MCP is enabled, it may also receive a small read-only control-plane tool set, but still receives no target-website client.

Deterministically validate the manager's sources

A prompt asking for citations is not a citation guarantee. Every successful route records its final URL in an application-owned registry. After kickoff, code checks that:

• Every specialist route recorded successful evidence.
• The report contains HTTPS citations.
• Every cited URL was observed by a specialist.
• At least one observed source from every route appears in the report.

This validates provenance, not truth. You still need domain-specific checks for extracted prices, dates, currencies, product IDs, claims, and conflicts.

Add Playwright for Browser-Heavy Pages

Use Playwright when the required content depends on client-side JavaScript, browser storage, or visible interaction. Do not add a browser merely because the workflow uses agents.

The Python Playwright proxy configuration belongs on Chromium launch:

Create a separate browser context per agent or work unit:

Locale and timezone can align the browser with an intended test scenario, but they do not prove the residential exit's location. Verify what the target actually displays.

The companion example includes an exact run_browser_smoke() path that:

• Launches Chromium with separated proxy fields.
• Uses a fresh context.
• Blocks service workers.
• Rejects non-HTTPS, non-port-443, and non-allowlisted resource requests.
• Validates the final navigation URL.
• Caps returned visible text.
• Reports target-visible marker matches without converting them into an unsupported IP claim.
• Closes the context and browser.

Run one route at a time:

HTTPX vs Playwright

Blocking images, media, fonts, analytics, or third-party requests can reduce bandwidth, but it may also change page behavior. Review and test the exact resource allowlist for every approved target.

Selenium follows the same boundary: configure its browser or driver proxy and keep one profile/session per work unit. CrewAI also documents packaged Selenium and hosted-browser tools, but confirm that a chosen integration supports your own external proxy and the isolation controls your workflow needs.

Sticky vs Rotating Sessions for CrewAI

Use Sticky behavior when several requests belong to one coherent observation. Use Random or rotating behavior when each request is independent and broader IP diversity matters more than continuity.

A Sticky session is not a permanent IP guarantee. Residential peers can disconnect, providers can enforce time limits, and network conditions can change. Design idempotent steps that can reacquire a route and revalidate the target-visible market.

The live validation for this guide used one supplied Random residential access point with the documented US country selector. Two geo services classified the observed exit as US, and two fresh HTTPX clients saw the same redacted exit fingerprint during the short test window. That does not turn a Random access point into a Sticky one and does not establish a session-duration guarantee. For the article's three-market production pattern, generate and validate three appropriate work-unit sessions first.

Rotate at clean boundaries

A useful lifecycle is:

Do not reuse one long-lived Sticky identity across unrelated customers, campaigns, or research subjects merely to avoid creating sessions. That weakens isolation and can carry cookies or reputation between tasks.

Geo-Target Each Agent Without Making Fake Claims

There are four different statements people often collapse into “the proxy is in Germany”:

1. The application intended to use Germany.
2. The access point was configured with a Germany selector.
3. An IP-geolocation service classified the observed exit as Germany.
4. The target website returned the intended Germany-specific experience.

Record them separately.

For a price agent, verify fields such as currency, tax treatment, regional store, shipping country, and product availability. For an SEO agent, verify the search engine domain, language, location parameter, and returned local result set. For a verification agent, compare the exact claim and source rather than treating a matching country code as sufficient.

The example hashes a successful IP preflight with a random, per-process HMAC key and logs only a 12-character fingerprint. That can show short-window continuity without exposing the raw exit address. It deliberately does not call that fingerprint a location.

*_EXPECTED_MARKERS provides a deterministic page-content check. If the configured markers are absent, the tool reports a mismatch. If they all appear, it reports that the markers matched—not that the IP is physically located in the intended country.

Keep Crew, Flow, Proxy, and Browser State Separate

CrewAI can preserve task context, memory, Flow state, and checkpoints. Those are not network leases.

If a Flow resumes after a crash, reacquire the correct proxy work unit, verify it, restore only approved browser state, and repeat an idempotent step if necessary. Do not assume a stored market="DE" field means the current network route is still Germany.

Do not put proxy passwords, complete proxy URLs, MCP tokens, raw exit IPs, or sensitive browser state into agent memory or checkpoint payloads. Store a non-secret route identifier and resolve it through an application-owned session manager or secret store.

Control Concurrency at Every Layer

Multi-agent execution creates several independent limits:

CrewAI documents Crew.max_rpm as overriding individual agents' max_rpm values when set. That limits model requests; it is not a substitute for an HTTP semaphore or per-target limiter inside the tool.

The controlled test used a synchronization barrier in an authenticated proxy fixture. All three specialist tools reached the proxy concurrently, proving that the async task arrangement actually overlaps the data-plane work. The manager remained synchronous and ran only after those tasks completed.

Bound retries independently

The example retries timeouts, transport failures, HTTP 429, and selected 5xx responses. It does not automatically retry every 4xx response or rotate the exit after each failure.

Good retry behavior includes:

• A small maximum attempt count.
• Exponential backoff with jitter.
• Idempotent operations.
• A shared deadline.
• Respect for an applicable Retry-After policy.
• No retry for invalid destinations, authentication failures, unsupported media, or deterministic policy rejection.
• A separate circuit breaker for sustained target failures.

A 403 can mean authorization, website policy, session state, request shape, or an access rule. Blindly changing IPs is not a diagnosis and can increase load.

Use Proxidize MCP as a Control Plane

CrewAI has first-class MCP support for stdio, SSE, Streamable HTTP, multiple servers, and tool filtering. The current docs recommend the native mcps field for most applications and retain MCPServerAdapter from crewai-tools for advanced manual connection management.

The Proxidize MCP server is published as @proxidize/mcp. Its tools depend on the subscriptions visible to the supplied account token.

Keep the two paths separate:

The native CrewAI configuration is:

Set PROXIDIZE_API_TOKEN to the fresh raw token. Do not include the literal Bearer prefix; the MCP package constructs authentication. A token is not the same secret as a residential proxy username/password.

Start with a tiny read-only tool allowlist. “Read-only” does not necessarily mean “safe to expose to a model”: a response schema may contain account, access-point, or credential data. Inspect the current tool schema and sanitize results before broadening access. Keep mutation tools out of the model-visible set unless an application-owned authorization and human-approval path has been designed and tested.

CrewAI's native static filter controls which discovered tools become available to the agent. It is not proof that the server token has least privilege, and it is not a human approval workflow. Enforce both server-side authorization and application-side policy.

The example keeps MCP disabled by default. The controlled validation proved CrewAI's native stdio connection and static filtering: it exposed and executed one harmless fixture status tool while withholding a fixture mutation tool. The live validation then connected to @proxidize/mcp 0.1.3, discovered the plan-scoped tool list, and completed get_subscription through both a direct stdio client and CrewAI's native MCP adapter. The CrewAI run exposed exactly one statically allowlisted live tool. No mutation ran, and neither account response content nor credentials were retained in the guide or test output.

Protect the Crew From Prompt Injection and SSRF

Any webpage can contain text designed to manipulate an agent. Treat it as data, even if it looks like a system message, tool instruction, security warning, or request for credentials.

Use layered controls:

1. Destination policy: exact HTTPS host allowlists, port restrictions, public-address checks, and redirect revalidation.
2. Tool scoping: each specialist gets only its route-specific fetch tool; the manager gets no target browser.
3. Secret isolation: credentials are private runtime fields, never tool arguments or prompt text.
4. Network isolation: deny access to metadata services, private networks, control planes, and unrelated hosts at the container or VPC layer as well as in code.
5. Content limits: accept only required media types and cap bytes and extracted text.
6. Action limits: bound agent iterations, tool calls, time, redirects, retries, and concurrency.
7. Output validation: require observed URLs and validate domain-specific structured fields.
8. MCP filtering: expose only necessary, sanitized control-plane tools.
9. Human review: require approval for state changes, sensitive operations, or high-impact decisions.
10. Safe logging: redact proxy credentials, API tokens, raw exits, cookies, authorization headers, and sensitive page content.

An application allowlist is necessary but not sufficient. Enforce egress policy outside the Python process so a library bug, browser feature, new tool, or prompt-driven code path cannot reach private infrastructure.

Complete Testable Example

The full source used for validation is embedded below and is also stored in the editorial package as crewai-agents-residential-proxies-example.py. It includes:

• Three route configurations and three private HTTPX clients.
• Three route-specific CrewAI tools with url as the only model-visible argument.
• Three concurrent specialist tasks and one synchronous manager task.
• Sticky-work-unit isolation primitives.
• HTTPS allowlists, public-address checks, redirect validation, response limits, retries, and concurrency controls.
• Redacted proxy-exit fingerprints.
• Target-visible marker reporting.
• Deterministic source validation.
• An optional Playwright smoke path.
• An optional, narrowly filtered native Proxidize MCP configuration.

Run the HTTPX Crew:

Then run at least one browser route:

Do not use --skip-preflight in a credentialed production check. It exists so controlled fixtures can test the Crew without pretending their local proxy has a public residential exit.

Live-Test Status

The code, live US residential route, real CrewAI model loop, browser path, and read-only Proxidize MCP path were tested on September 2, 2026. The validation was intentionally limited to the United States and one Random access point.

The requested US-only live scope is complete. The supplied access point was Random, so the test used the documented US selector without relabeling it Sticky. Reusing that credential across three agents proves the real model, HTTPX, Playwright, tool, manager, citation, and client-isolation paths; it does not prove three provider-side sessions or three geographies.

No raw exit IP, model key, proxy credential, MCP token, account response, or complete authenticated proxy URL is included in this guide or its companion source. Because the test credentials had already been shared in chat, rotate them before further use.

Common Mistakes

Putting the proxy on the model client

That changes model-provider routing, not necessarily the custom website tool. Configure the HTTP or browser client that makes the target request.

Using one random endpoint for three market claims

A random route can prove authentication and page access. It cannot prove a Sticky US, UK, and Germany configuration. Generate and validate separate work-unit routes.

Sharing one client across all specialists

This can mix cookies, connections, cache, exit assignment, target state, evidence, and rate limits. Use separate route objects and clients.

Calling a configured location “verified”

A selector expresses intent. Verify the target-visible outcome and preserve uncertainty when signals disagree.

Treating async_execution=True as a complete concurrency policy

It schedules overlapping Crew tasks. It does not limit HTTP sockets, browser processes, per-host request rate, proxy bandwidth, or target load.

Assuming max_rpm limits page requests

CrewAI's RPM setting controls model requests. Put a semaphore and target-aware limiter inside the network layer.

Treating Crew memory as a proxy session

Memory and checkpoints do not restore an exit lease, browser context, or cookies. Reacquire and verify network state when resuming.

Following redirects automatically

A permitted starting URL can redirect to an unapproved or private destination. Disable automatic redirects and validate each hop.

Letting the model construct proxy credentials

Keep dashboard-generated selectors and passwords opaque. Resolve an approved non-secret route ID inside application code.

Calling every failure an IP problem

403, 429, 5xx, parsing failures, and target changes have different causes. Inspect status, headers, target policy, session state, and application logs before rotating.

Exposing every MCP tool

Tool discovery is not an authorization design. Use a tiny allowlist, sanitize responses, and keep mutation tools behind independent approval.

Logging full URLs indiscriminately

Target URLs may contain sensitive query values, and proxy URLs contain credentials. Log normalized, redacted fields and approved evidence identifiers.

Troubleshooting

Conclusion

The clean CrewAI architecture is one proxy-aware tool per network-facing work unit. Let CrewAI coordinate specialist tasks and manager context; let HTTPX or Playwright carry website traffic through separately generated Proxidize Residential sessions; and treat Proxidize MCP as an optional, tightly filtered control plane.

For a three-market Crew, use isolated Sticky sessions for coherent US, UK, and Germany observations, then verify what each target actually displays. Bound Crew loops separately from HTTP and browser concurrency, keep state layers independent, and validate citations and structured evidence outside the model.

Explore Proxidize Residential Proxies, read Proxy Sessions for AI Agents, compare cloud browsers for AI agents, or review the related OpenAI Agents SDK, LangGraph, Claude Browser Use, Google ADK, and Microsoft Agent Framework integration guides.