4 Best Python Libraries for Web Scraping in 2026

January 24, 2025

Developers are always looking for the best tools for the job. Nowadays the internet is full of resources and libraries that you can use to optimise your work. Unfortunately, in our enthusiasm to try the latest tools and technologies, devs can jump the gun. Many tools will be still in early stages or still not mature enough to handle complex use cases, which leaves us disappointed.

In this article we’re going to talk about the best Python libraries for web scraping. Python is one of the most used programming languages in the world for web scraping. It has rich libraries and a very large community. If you are using AI to code, Python is generally the go-to language for AI.

Let’s take a real example from a fellow developer. Let’s say you work at a company that does an intelligent pricing for books and sells the data as a service. Say you were tasked to create a Python script that scrapes this website and gets the following data: Book name, price, and availability. Your script has to save that data to be sold for the users later.

High-quality scraping and automation starts with high-quality mobile proxies

After a meeting with stakeholders you eventually go look at the website and see how its HTML is structured. You start to put together a plan for how to collect the data you need. You begin looking for the best tool for the job, which leads you down the rabbit hole of Python libraries.

Let’s talk about the four best Python libraries for web scraping. We’ll walk you through code examples, when to use which libraries, and each one’s pros and cons so that by the end you will have a better understanding of how to use each one and how to make your code more efficient/effective as well.

Best Practices Before Web Scraping

Before you start picking a Python web scraping library, there are some steps you should take before you start coding.

Check The Website Structure

As a developer you should always check the website you are scraping. By that I mean looking at the inspect window and reading the target site’s code. Pay attention to how it structures its data so you can build your code in function of the website.

Always Use a Virtual Environment in Python

To get started with these libraries, you need to install them. This is where some devs go wrong by falling into the trap of downloading each package every time you start a new project. This is a great way to run out of space.

To solve this you need to use something called venv (virtual environment variable). It creates a virtual environment for just the current project and won’t affect any other projects on your device. This both saves you space and keeps projects isolated from others.

Use a Package Manager

Using a package manager can make your life easier as a developer. You can control all your packages in one place. For this article, we used uv. It’s easy to use, fast, and user friendly.

Now that we’ve covered the pre-project checks, let’s dive into the four best Python libraries for web scraping.

BeautifulSoup4

BeautifulSoup is a library used to scrape information from web pages. Essentially it sits on top of an HTML or XML parser providing idioms for iteration, searching and modifying the parse tree. This library has a great deal of credibility because it has been around for the past 20 years. It’s very mature and well maintained.

Installing BeautifulSoup4

Installing the library is as simple as installing any other library; simply copy and paste the command into your terminal and keep in mind the recommended download techniques we discussed.

pip install beautifulsoup4

pip install beautifulsoup4

Web Scraping with BeautifulSoup4

Once installed — to continue our example web scraping task — we will try to scrape the pricing, book name, and availability of the books of our target website. Requests is a required library to web scrape with BeautifulSoup4 to process HTTP requests.

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
response = requests.get(url)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

books = soup.select("article.product_pod")

scraped_data = []

for book in books:
    title = book.select_one("h3 a")["title"]
    price = book.select_one("p.price_color").text
    availability = book.select_one("p.instock.availability").text.strip()

    scraped_data.append({
        "title": title,
        "price": price,
        "availability": availability
    })

for idx, book in enumerate(scraped_data, start=1):
    print(f"{idx}. {book['title']} — {book['price']} — {book['availability']}")

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
response = requests.get(url)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

books = soup.select("article.product_pod")

scraped_data = []

for book in books:
    title = book.select_one("h3 a")["title"]
    price = book.select_one("p.price_color").text
    availability = book.select_one("p.instock.availability").text.strip()

    scraped_data.append({
        "title": title,
        "price": price,
        "availability": availability
    })

for idx, book in enumerate(scraped_data, start=1):
    print(f"{idx}. {book['title']} — {book['price']} — {book['availability']}")

Pros of Web Scraping with BeautifulSoup4

Beginner friendly: If you are just starting as a developer, you will be in safe hands with this library. It’s one of the most beginner friendly web scraping libraries out there with clear documentation and API.
Strong community support: Because it’s been around for the last 20 years, many people have used it and built projects on top of it. That means there’s a lot of people online to ask for help if you need it.
Lightweight and fast: For static pages, BeautifulSoup is faster than any library, because its main strength is to scrape HTML websites. Most of the time when dealing with this kind of website you don’t need JavaScript execution.

Cons of Web Scraping with BeautifulSoup4

Requires external libraries: BeautifulSoup is a parsing library. It lacks its own way to handle requests or do any automation scripts, so it needs other libraries as we saw earlier in the code.
Relatively slower than other libraries: It uses Python’s built-in html.parser which is slower compared to other alternatives such as lxml. It supports synchronous operations by default, which makes it slow, since it waits for the other request to finish.

You always should maintain the code: Because BeautifulSoup relies on HTML structure, you will have to update your code whenever the website’s structure changes.

Scrapy

Scrapy is one of the newest Python libraries for web scraping available out there. It has a very good reputation and developers love it because it’s both free and open source. It can take care of everything for you, including managing requests, storing data in an organized way to make it easier for you to start scraping.

Installing Scrapy

Scrapy’s website is cool and offers very clear documentation and an extremely simple installation process. Basically, you copy and paste the command into your terminal and you are ready to start scraping.

pip install scrapy

pip install scrapy

Web Scraping with Scrapy

Now you should be able to start scraping the book website and extract the data needed from it.

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for book in response.css("article.product_pod"):
            availability = "".join(book.css("p.instock.availability::text").getall()).strip()
            
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "availability": availability
            }

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for book in response.css("article.product_pod"):
            availability = "".join(book.css("p.instock.availability::text").getall()).strip()
            
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css("p.price_color::text").get(),
                "availability": availability
            }

Pros of Web Scraping with Scrapy

High performance and efficiency: Scrapy is built on asynchronous networking, which allows it to handle thousands of requests concurrently without the need to wait for other requests. This makes it highly efficient and desirable to developers.
Built-in features: It contains many features that make the life of a developer easier, like support for CSS selectors and XPath expressions, and it has an autothrottle feature that prevents overloading websites.
Multiple output formats: It’s a nice feature to have, especially when you have to work with data or AI frequently. Being able to have your collected data in a variety formats can save you a lot of time and effort.

Cons of Web Scraping with Scrapy

Overkill for small projects: Scrapy is great, but to really take full advantage of the Python library you should use it for larger projects. That’s where you’ll also find you have a need to make use of its many features.
Steep learning curve: Scrapy can be a lot for a first-time user or junior dev. You’ll get used to it but it’ll take some time.

Playwright

Playwright is an open-source framework that is backed and developed by Microsoft. It enables you to automate multiple browsers, like Chromium and Firefox, in a single API. The Playwright library is used to do automated tasks in browsers.

Installing Playwright

To install the framework, it’s a straightforward process. Like the other libraries we used, we will use pip as our installation manager.

pip install pytest-playwright

pip install pytest-playwright

Web Scraping with Playwright

After installation, you will be able to start automating. You’ll also need a few other libraries like chromium and asyncio.

import asyncio
import json
from playwright.async_api import async_playwright

async def scrape_books():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        await page.goto("https://books.toscrape.com/")
        await page.wait_for_selector("article.product_pod")

        books = []

        book_elements = await page.query_selector_all("article.product_pod")

        for book in book_elements:
            title = await book.query_selector_eval(
                "h3 a", "el => el.getAttribute('title')"
            )
            price = await book.query_selector_eval(
                "p.price_color", "el => el.textContent"
            )
            availability = await book.query_selector_eval(
                "p.instock.availability",
                "el => el.textContent.replace(/\\s+/g, ' ').trim()"
            )

            books.append({
                "title": title,
                "price": price,
                "availability": availability
            })

        await browser.close()
        return books


async def main():
    books = await scrape_books()

    print(f"Scraped {len(books)} books\n")
    print(json.dumps(books, indent=2, ensure_ascii=False))

    with open("books_playwright.json", "w", encoding="utf-8") as f:
        json.dump(books, f, indent=2, ensure_ascii=False)

    print("Data saved to books_playwright.json")


if __name__ == "__main__":
    asyncio.run(main())

import asyncio
import json
from playwright.async_api import async_playwright

async def scrape_books():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()

        await page.goto("https://books.toscrape.com/")
        await page.wait_for_selector("article.product_pod")

        books = []

        book_elements = await page.query_selector_all("article.product_pod")

        for book in book_elements:
            title = await book.query_selector_eval(
                "h3 a", "el => el.getAttribute('title')"
            )
            price = await book.query_selector_eval(
                "p.price_color", "el => el.textContent"
            )
            availability = await book.query_selector_eval(
                "p.instock.availability",
                "el => el.textContent.replace(/\\s+/g, ' ').trim()"
            )

            books.append({
                "title": title,
                "price": price,
                "availability": availability
            })

        await browser.close()
        return books


async def main():
    books = await scrape_books()

    print(f"Scraped {len(books)} books\n")
    print(json.dumps(books, indent=2, ensure_ascii=False))

    with open("books_playwright.json", "w", encoding="utf-8") as f:
        json.dump(books, f, indent=2, ensure_ascii=False)

    print("Data saved to books_playwright.json")


if __name__ == "__main__":
    asyncio.run(main())

Pros of Web Scraping with Playwright

It’s open source: Developers love open-source projects. We love to be in complete control of our code and Playwright gives us the freedom to modify our code as we see fit.
Automatic waits: If you’ve ever dabbed in web scraping before, you know how annoying it is to create multiple tests to make sure that waits are visible. I struggled with this when I was building a Twitter/X scraper until I switched to Playwright. It creates its own automatic testing, which can save you a significant amount of time.
Headless or GUI mode: Speed is another thing devs care about. Achieving speed isn’t possible without trial, errors, and debugging. Luckily, Playwright solves this problem by giving you the option of using GUI mode for debugging and testing your code. Then, when you need speed for functional code, you switch to headless mode.

Cons of Web Scraping with Playwright

Learning curve: Playwright isn’t difficult to learn, but it will take some time for a developer unfamiliar with it to get used to it. The same is true of using any new tool for the first time.
Large installation size: Playwright has a few prerequisites that need to be installed before you get started, and these files can be resource intensive sometimes.
Small community: Although it’s growing and more people are switching to Playwright, it still has a very small community compared to something like Selenium. That means there will be fewer people around to help you if you really hit a wall.

Selenium

Selenium is an open-source framework and it’s one of the oldest open-source projects there is. It has a very large community and lots of support. It allows you to automate browser tasks.

Installing Selenium

To install the last version of the framework, you should add the following command to your terminal:

pip install selenium

pip install selenium

Web Scraping with Selenium

After the installation you can start web scraping with Selenium via your code:

import json
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager


def scrape_books():
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")

    print("Initializing Chrome WebDriver...")
    service = Service(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=service, options=chrome_options)

    try:
        print("Navigating to https://books.toscrape.com/")
        driver.get("https://books.toscrape.com/")

        wait = WebDriverWait(driver, 10)

        wait.until(EC.title_contains("Books"))
        wait.until(EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, "article.product_pod")
        ))

        book_elements = driver.find_elements(
            By.CSS_SELECTOR, "article.product_pod"
        )

        books_data = []

        for book in book_elements:
            title = book.find_element(
                By.CSS_SELECTOR, "h3 a"
            ).get_attribute("title")

            price = book.find_element(
                By.CSS_SELECTOR, "p.price_color"
            ).text

            availability = " ".join(
                book.find_element(
                    By.CSS_SELECTOR, "p.instock.availability"
                ).text.split()
            )

            books_data.append({
                "title": title,
                "price": price,
                "availability": availability
            })

        return books_data

    finally:
        driver.quit()


def main():
    print("Starting Selenium scraper...\n")

    books = scrape_books()

    print(f"Scraped {len(books)} books:\n")
    print(json.dumps(books, indent=2, ensure_ascii=False))

    with open("books_selenium.json", "w", encoding="utf-8") as f:
        json.dump(books, f, indent=2, ensure_ascii=False)

    print("\n✓ Data saved to books_selenium.json")


if __name__ == "__main__":
    main()

import json
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager


def scrape_books():
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")

    print("Initializing Chrome WebDriver...")
    service = Service(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=service, options=chrome_options)

    try:
        print("Navigating to https://books.toscrape.com/")
        driver.get("https://books.toscrape.com/")

        wait = WebDriverWait(driver, 10)

        wait.until(EC.title_contains("Books"))
        wait.until(EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, "article.product_pod")
        ))

        book_elements = driver.find_elements(
            By.CSS_SELECTOR, "article.product_pod"
        )

        books_data = []

        for book in book_elements:
            title = book.find_element(
                By.CSS_SELECTOR, "h3 a"
            ).get_attribute("title")

            price = book.find_element(
                By.CSS_SELECTOR, "p.price_color"
            ).text

            availability = " ".join(
                book.find_element(
                    By.CSS_SELECTOR, "p.instock.availability"
                ).text.split()
            )

            books_data.append({
                "title": title,
                "price": price,
                "availability": availability
            })

        return books_data

    finally:
        driver.quit()


def main():
    print("Starting Selenium scraper...\n")

    books = scrape_books()

    print(f"Scraped {len(books)} books:\n")
    print(json.dumps(books, indent=2, ensure_ascii=False))

    with open("books_selenium.json", "w", encoding="utf-8") as f:
        json.dump(books, f, indent=2, ensure_ascii=False)

    print("\n✓ Data saved to books_selenium.json")


if __name__ == "__main__":
    main()

Pros of Web Scraping with Selenium

It’s open source: As stated before, developers love open-source projects because they can tinker with the codebase until it exactly fulfills their requirements.
Large community: Since it’s a framework that has been available for just over two decades, many people have used it and contributed to it. That gives it an advantage to many comparable frameworks: a lot of people have solved many different problems with Selenium. The chances that you’re trying to accomplish something truly unique with it are small, and you can fall back on an experienced community for help.
Broadly supported: Selenium works perfectly on many platforms, from Mac, Windows, and Linux to most browsers, like Google Chrome, Firefox, and Opera.

Cons of Web Scraping with Selenium

Learning curve: A long-lived library comes with many updates and a learning curve. Teams that try to avoid maintenance try to avoid Selenium for this reason.
Lacks built-in capabilities: Selenium is great, but libraries that come with built-in tools are better and unfortunately Selenium doesn’t offer that. There are many cases in which you’ll need a third-party tool to do some tasks, which costs money and time.

Python Libraries for Web Scraping Compared: Practical Test

The easiest and fastest way to compare these four Python libraries for web scraping to each other is to have them each scrape the same target website and compare the results. For each, I write some simple code. Each of them had to do the following:

Open the website
Scrape the website and retrieve the title, price, and availability of the book
Return the output in JSON format

This is a very simple test, one that admittedly doesn’t truly contrast the complexities of each library’s features, but it can provide us with an indication of which one is easier and faster to use.

Output

Each library outputs the same data. We scraped the website with the intent of getting the title, price and availability of each book.

[
  {
    "title": "A Light in the Attic",
    "price": "\u00a351.77",
    "availability": "In stock"
  },
  {
    "title": "Soumission",
    "price": "\u00a350.10",
    "availability": "In stock"
  },
  {
    "title": "Sharp Objects",
    "price": "\u00a347.82",
    "availability": "In stock"
  },
  {
    "title": "Sapiens: A Brief History of Humankind",
    "price": "\u00a354.23",
    "availability": "In stock"
  },
  {
    "title": "The Requiem Red",
    "price": "\u00a322.65",
    "availability": "In stock"
  },
  {
    "title": "The Dirty Little Secrets of Getting Your Dream Job",
    "price": "\u00a333.34",
    "availability": "In stock"
  },
]

[
  {
    "title": "A Light in the Attic",
    "price": "\u00a351.77",
    "availability": "In stock"
  },
  {
    "title": "Soumission",
    "price": "\u00a350.10",
    "availability": "In stock"
  },
  {
    "title": "Sharp Objects",
    "price": "\u00a347.82",
    "availability": "In stock"
  },
  {
    "title": "Sapiens: A Brief History of Humankind",
    "price": "\u00a354.23",
    "availability": "In stock"
  },
  {
    "title": "The Requiem Red",
    "price": "\u00a322.65",
    "availability": "In stock"
  },
  {
    "title": "The Dirty Little Secrets of Getting Your Dream Job",
    "price": "\u00a333.34",
    "availability": "In stock"
  },
]

Speed Comparison

As you can see from the results, BeautifulSoup4 is the winner here. It was the easiest and the fastest to set up between all four, and is specifically designed for static, easy-to-scrape sites.

Library	Setup Difficulty	Average Time	Difference
BeautifulSoup4	Very easy	0.896s	Fastest
Scrapy	Medium	0.910s	+0.014s
Playwright	Easy	2.330s	+1.434s
Selenium	Medium	1.671s	+0.775s

Scrapy was only a fraction of a second behind BeautifulSoup, with the others following shortly after.

Conclusion

When looking for a Python library for web scraping, you have a lot of options to choose from as a developer. We hope this article has made that choice easier. Remember to communicate with your stakeholders to understand their requirements and needs. Let those inform what choice of web scraping library.

Key takeaways:

Always have a clear set of requirements in mind before you start any web scraping project. Choose your tech stack in function of the project’s goals.
If you have a small project and limited budget, choose BeautifulSoup, since it’s fast and great fit for small projects.
If you care about speed and efficiency, choose Scrapy, a very fast library that has a lot of built-in features that will save you time and money.
Always use venv for installing Python packages; you don’t want to run out of space and affect other projects on your device.

Although there are other Python libraries for web scraping than are mentioned in this article, these are the four that stand out from the perspective of a developer. Regardless of the project’s scale or scope, choosing the right web scraping library can make all the difference.

A library with a large community will likely be able to offer broader support; that might make your life easier. Treading new ground is always more demanding than being able to borrow solutions from people who have already solved problems you’re facing.