Web Scraping with Selenium and Python

July 19, 2024

Introduction to Selenium

Selenium is an open-source framework designed for automating web browsers. While it was initially developed for testing web applications, it evolved to become a robust tool for web automation, making it a popular choice for web scraping. Its ability to interact with web pages just like a human user sets it apart from other scraping tools, allowing for complex interaction with dynamic content.

Key Features of Selenium

Cross-Browser Support: Selenium supports multiple browser windows, including Chrome, Firefox, Safari, and Edge, enabling you to test and scrape across different environments.
Multiple Language Bindings: Selenium offers bindings for various programming languages, such as Python, Java, C#, Ruby, and JavaScript, providing flexibility for developers.
Plugin Capability: Selenium can be extended with plugins and integrations, allowing for customization and enhancement of its capabilities.
Community and Support: Being open-source, Selenium has a large community, extensive documentation, and numerous online resources to assist users.

Components of Selenium

Selenium is composed of several key components that work together to facilitate web automation:

Selenium WebDriver: The core component of Selenium is WebDriver. It interacts with web browsers to perform actions such as clicking buttons, filling forms, and navigating web pages. It provides a programming interface to control the browser and execute automated tasks.
Selenium IDE (Integrated Development Environment): Selenium IDE is a browser extension that allows users to record and playback interactions with web pages. It is useful for creating quick test cases and prototyping.
Selenium Grid: Selenium Grid allows for parallel test execution across multiple machines and browsers, enabling efficient large-scale testing and scraping operations. It distributes tests across different environments, reducing execution time and improving coverage.

Why Do Web Scraping with Selenium?

There are several advantages that make web scraping with Selenum an excellent choice. Its ability to handle dynamic content is particularly notable, as many modern websites use JavaScript to load elements dynamically. Selenium can wait for these elements to load before extracting data, ensuring accurate scraping results. It excels in interacting with complex web elements, such as dropdowns, modal windows, and infinite scrolls, allowing for more comprehensive data extraction compared to traditional scraping tools. The tool’s ability to simulate user actions, like mouse clicks, keyboard input, and form submissions, is crucial for automating tasks that require user interaction, such as scraping data behind login forms or performing multi-step processes.

Selenium’s support for multiple browsers and platforms ensures compatibility across various environments, providing versatile scraping solutions. Its robust error handling and debugging capabilities make it easier to identify and resolve issues during the scraping process which enhances reliability. Its integration with other tools and libraries, such as BeautifulSoup for HTML parsing and pandas for data manipulation, further extends its functionality and versatility, making it a powerful and flexible choice for web scraping projects.

How to Install Selenium in Python

Prerequisites

Before you begin web scraping with Selenium, you’ll need to ensure you have the necessary software and tools installed:

Python: Selenium supports various programming languages, but Python is a popular choice due to its simplicity and extensive libraries. For this article, we will be providing code examples written in Python.
pip: This is Python’s package installer, which you’ll use to install Selenium. It usually comes bundled with Python.
Web Browser: Selenium works with multiple browsers, but starting with Google Chrome is often recommended due to its widespread use and robust support.
ChromeDriver: This is a separate executable that Selenium WebDriver uses to control Chrome. You can download the appropriate version of ChromeDriver from the ChromeDriver download page. To find which version to download, check the version of Chrome you have and download the correct one.

How to Install Selenium

Follow these steps to set up Selenium and its dependencies:

Install Selenium: Open your terminal or command prompt and run the following command to install Selenium using pip:

pip install selenium

Download ChromeDriver: Visit the ChromeDriver download page and download the version that matches your Chrome browser version. Extract the downloaded file to a location on your system and note the path.
Add ChromeDriver to PATH: Add the path to the ChromeDriver executable to your system’s PATH environment variable. This step ensures that you can launch ChromeDriver from any directory.

Now that you have everything set up, let’s write a simple Selenium script to open a webpage and extract basic data.

Open your text editor or IDE and create a new Python file.
Import the necessary modules and set up the WebDriver for Chrome:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

Use Selenium to open a webpage. For this example, we’ll open the Python official website:

# Open the Python official website
driver.get('https://www.python.org')

Extract some basic information from the webpage, such as the title:

# Get the title of the webpage
title = driver.title
print(f'Title: {title}')

Finally, close the browser once you have extracted the data:

# Close the browser
driver.quit()

Here is what your full script should look like:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Get the title of the webpage
title = driver.title
print(f'Title: {title}')

# Close the browser
driver.quit()

Run this script in your terminal or command prompt:

python first_selenium_script.py

A drawing of a person works on a screen under the title

Data Extraction and Storage

Extracting Data using Selenium in Python

Selenium provides robust tools to accomplish data extraction. Here’s how to extract data effectively using Selenium in Python.

Locating Elements:

Selenium offers various methods to locate elements on a web page, such as by ID, class name, tag name, name attribute, link text, partial link text, CSS selector, and XPath. Here are some examples:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Get the title of the webpage
title = driver.title
print(f'Title: {title}')

# Locate elements using the updated method
element_by_id = driver.find_element(By.ID, 'element_id')
element_by_class = driver.find_element(By.CLASS_NAME, 'element_class')
element_by_css = driver.find_element(By.CSS_SELECTOR, '.element_class')
element_by_xpath = driver.find_element(By.XPATH, '//tag[@attribute="value"]')

# Extracting text and attribute
element_text = element_by_id.text
print(f'Text: {element_text}')

element_attribute = element_by_id.get_attribute('href')
print(f'Attribute: {element_attribute}')

# Extracting multiple elements
elements = driver.find_elements(By.CLASS_NAME, 'element_class')
for element in elements:
    print(element.text)

# Close the browser
driver.quit()

Extracting Text and Attributes:

Once you’ve located the elements, you can extract the desired data, such as text content or attributes:

# Extracting text
element_text = element_by_id.text
print(f'Text: {element_text}')

# Extracting attribute
element_attribute = element_by_id.get_attribute('href')
print(f'Attribute: {element_attribute}')

Extracting Multiple Elements:

To extract data from multiple elements, you can use methods like `find_elements_by` which return a list of elements:

# Locate multiple elements using the updated method
elements = driver.find_elements(By.CLASS_NAME, 'element_class')
for element in elements:
    print(element.text)

Here is an example script scraping Python’s news section:

First, you must ensure that you install BeautifulSoup into your code by writing:

pip install selenium beautifulsoup4

After that is done, here is what the full code would look like:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Extract the page source and parse it with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

# Find the news section
news_section = soup.find('div', class_='medium-widget blog-widget')

# Find all list items in the news section
news_items = news_section.find_all('li')

# Extract the date and title for each news item
for item in news_items:
    date = item.find('time').get('datetime')
    title = item.find('a').text
    print(f'Date: {date}, Title: {title}')

# Close the browser
driver.quit()

Data Storage Options

Once the data is extracted, storing it efficiently is crucial for further analysis and usage. Here are some common data storage options:

CSV Files:

CSV (Comma-Separated Values) files are simple and widely used for storing tabular data. They can be easily opened in spreadsheet applications like Excel. Here’s how to save data to a CSV file using Python’s `csv` module:

import csv
with open('data.csv', 'w', newline='') as csvfile:
    fieldnames = ['date', 'title']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for row in data:
        writer.writerow(row)

The full script will look like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import csv

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Extract the page source and parse it with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

# Find the news section
news_section = soup.find('div', class_='medium-widget blog-widget')

# Find all list items in the news section
news_items = news_section.find_all('li')
data = []
# Extract the date and title for each news item
for item in news_items:
    date = item.find('time').get('datetime')
    title = item.find('a').text
    print(f'Date: {date}, Title: {title}')
    data.append(
        {
            "date": date,
            "title": title
        }
    )

with open('data.csv', 'w', newline='') as csvfile:
    fieldnames = ['date', 'title']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    for row in data:
        writer.writerow(row)
# Close the browser
driver.quit()

JSON Files:

JSON (JavaScript Object Notation) is a lightweight data-interchange format that is easy for humans to read and write, and easy for machines to parse and generate. Here’s how to save data to a JSON file using Python’s `json` module:

import json

# Save data to JSON file
with open('data.json', 'w') as jsonfile:
    json.dump(data, jsonfile, indent=4)

The full code would look like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import json

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'
service = Service(driver_path)

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Extract the page source and parse it with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

# Find the news section
news_section = soup.find('div', class_='medium-widget blog-widget')

# Find all list items in the news section
news_items = news_section.find_all('li')
data = []
# Extract the date and title for each news item
for item in news_items:
    date = item.find('time').get('datetime')
    title = item.find('a').text
    print(f'Date: {date}, Title: {title}')
    data.append(
        {
            "date": date,
            "title": title
        }
    )

# Save data to JSON file
with open('data.json', 'w') as jsonfile:
    json.dump(data, jsonfile, indent=4)

# Close the browser
driver.quit()

Databases:

For larger datasets or more complex data structures, databases are a fantastic solution. SQLite, MySQL, and PostgreSQL are common choices. Here’s how to save data to an SQLite database using Python’s `sqlite3` module:

import sqlite3

# Save data to SQLite database
conn = sqlite3.connect('data.db')
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS news (date TEXT, title TEXT)''')
c.executemany('INSERT INTO news (date, title) VALUES (?, ?)', [(item['date'], item['title']) for item in data])
conn.commit()
conn.close()

The full code would look like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import sqlite3

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Extract the page source and parse it with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

# Find the news section
news_section = soup.find('div', class_='medium-widget blog-widget')

# Find all list items in the news section
news_items = news_section.find_all('li')
data = []
# Extract the date and title for each news item
for item in news_items:
    date = item.find('time').get('datetime')
    title = item.find('a').text
    print(f'Date: {date}, Title: {title}')
    data.append(
        {
            "date": date,
            "title": title
        }
    )

# Save data to SQLite database
conn = sqlite3.connect('data.db')
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS news (date TEXT, title TEXT)''')
c.executemany('INSERT INTO news (date, title) VALUES (?, ?)', [(item['date'], item['title']) for item in data])
conn.commit()
conn.close()

# Close the browser
driver.quit()

NoSQL Databases:

For unstructured or semi-structured data, NoSQL databases like MongoDB are ideal. Here’s how to save data to MongoDB using the `pymongo` library:

from pymongo import MongoClient

# Save data to MongoDB NoSQL database
client = MongoClient('mongodb://localhost:27017/')
db = client['scraping_database']
collection = db['news']
collection.insert_many(data)

The full code will look like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
from pymongo import MongoClient

# Specify the path to the ChromeDriver executable
driver_path = 'path/to/chromedriver'

# Initialize the Chrome WebDriver using the Service object
driver = webdriver.Chrome(service=service)

# Open the Python official website
driver.get('https://www.python.org')

# Extract the page source and parse it with BeautifulSoup
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

# Find the news section
news_section = soup.find('div', class_='medium-widget blog-widget')

# Find all list items in the news section
news_items = news_section.find_all('li')
data = []
# Extract the date and title for each news item
for item in news_items:
    date = item.find('time').get('datetime')
    title = item.find('a').text
    print(f'Date: {date}, Title: {title}')
    data.append(
        {
            "date": date,
            "title": title
        }
    )

# Save data to MongoDB NoSQL database
client = MongoClient('mongodb://localhost:27017/')
db = client['scraping_database']
collection = db['news']
collection.insert_many(data)

# Close the browser
driver.quit()

Important Note: Replace the driver_path with the destination of your ChromeDriver file. After doing so, write “r” before the destination so that the code will not read the “\” as part of the text. Example: driver_path = r‘C:\Users\(user)\Desktop\(file name)\chromedriver.exe’

By effectively extracting data with Selenium and storing it using these various methods, you can ensure that your web scraping efforts result in valuable and easily accessible datasets. This allows for further analysis, reporting, and application of the scraped data in meaningful ways.

Selenium in Python Best Practices

Respecting Robots.txt

When scraping websites, it’s essential to respect the site’s `robots.txt` file. This file contains rules and guidelines that specify which parts of the site can be accessed by web crawlers and which parts should be avoided. Ignoring these rules can lead to your IP address being banned from accessing the site. A great way to go around your IP being banned is by using a proxy server, which will be discussed in more detail later on.

Understanding Robots.txt:

The `robots.txt` file is located in the root directory of a website (e.g., `https://example.com/robots.txt`). It contains directives like `Allow` and `Disallow` that indicate which pages or directories can be crawled.

User-agent: *
Disallow: /private/

Checking Robots.txt:

Before scraping a site, always check its `robots.txt` file to understand the rules. You can do this manually or automate the process using a Python library like `robotparser`.

import urllib.robotparser
rp = urllib.robotparser.RobotFileParser()
rp.set_url('https://example.com/robots.txt')
rp.read()
if rp.can_fetch('*', 'https://example.com/path/to/page'):
    print("Allowed to scrape this page")
else:
    print("Disallowed to scrape this page")

Rate Limiting

Rate limiting is crucial to avoid overloading the target website’s server with too many requests in a short period. Sticking to rate limits ensures that your scraping activity remains undetected and minimizes the risk of being banned.

Implementing Delays:

Introduce delays between requests to mimic human browsing behavior. This can be done using the `time.sleep` function.

import time

for url in urls_to_scrape:
    # Your scraping code here
    time.sleep(2)  # Wait for 2 seconds before the next request

Randomized Delays:

Use randomized delays to make the scraping activity even less detectable and more human-like.

import time
import random

for url in urls_to_scrape:
    # Your scraping code here
    time.sleep(random.uniform(1, 3))  # Wait for a random time between 1 to 3 seconds

Maintaining Anonymity

Maintaining anonymity is essential to protect your identity and avoid IP bans. Using mobile proxies is an effective way to achieve this. They route traffic through mobile networks, making the traffic appear to come from mobile devices. This can be particularly useful for accessing mobile-optimized sites or avoiding restrictions applied to desktop IP addresses. Below, you will find the code necessary to apply a mobile proxy to your web scraping efforts.

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': 'mobile_proxy_ip:port',
    'sslProxy': 'mobile_proxy_ip:port'
})
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
driver = webdriver.Chrome(desired_capabilities=capabilities, executable_path='path/to/chromedriver')

By adhering to these best practices, you can ensure that your web scraping activities are respectful, efficient, and discreet. Respecting `robots.txt` guidelines, implementing rate limiting, and maintaining anonymity through the use of proxies will help you scrape data responsibly and sustainably.

Conclusion

In this article, we’ve laid out the basics of web scraping with Selenium and Python and how to install Selenium in Python. We’ve also discussed how to extract data with Selenium and then store it in a variety of formats once it’s been extracted. Additionally, we covered the importance of adhering to best practices, such as respecting `robots.txt` guidelines, implementing rate limiting, and maintaining anonymity through the use of proxies. We hope that this guide has given you enough information to be able to dip your toes into web scraping with Selenium yourself.