How to Scraping Dynamic Websites with Python?

In this data-driven era, our thirst for information is almost infinite. However, with the rapid development of front-end technologies, traditional static scraping methods often fall short when faced with modern web pages. Have you ever encountered the confusion where the video titles and views visible in the browser come out blank when scraped with Python? This is known as “dynamically loaded content.” To tackle this problem, today we will delve into how to perform dynamic website scraping using Python, leveraging Selenium as a powerful tool to accurately extract the data we need from complex platforms like YouTube.

Why Static Scraping Fails in the Face of Dynamic Web Pages?

Before discussing how to operate specifically, let’s first clarify the essence of the problem. When we talk about web scraping dynamic content, we are facing a DOM structure that is generated in real time by JavaScript.

● Static Scraping (Requests/Beautiful Soup): It’s like taking a photograph. It only captures what the server sends you in that instant. If the data is loaded later, there’s naturally nothing in the photo.

● Dynamic Scraping (Selenium): It’s like sending a robot to observe on-site. Selenium truly launches a browser, executing JavaScript like a real person, waiting for the data to load completely before it starts collecting information.

If you are trying dynamic website scraping using Python, you will find that the advantage of Selenium is that it is not just a library; it is an automation engine that can simulate clicks, scrolls, and even handle complex login validations, making it the preferred tool for dealing with modern web pages.

Understanding Python Library: Selenium

Selenium is a Python library for browser automation control. It does not directly request webpage source code, but instead drives a real browser to load pages, execute JavaScript, and read the DOM content after the page is fully rendered.

When scraping YouTube, information such as the video list and views is dynamically generated after the page loads, and only a browser environment can fully present this data. For this reason, this article chooses Selenium as the operational tool.

Since you have a clear understanding of how Selenium works, let’s first set up a stable and reliable runtime environment locally.

Setting Up the Selenium Development Environment

Before starting to scrape dynamic websites, we need to set up the Python environment and the corresponding WebDriver.

1. Install the Selenium library

Run the following command in your terminal:

pip install selenium

2. Start the WebDriver

Selenium requires a "middleman" to control the browser. For Chrome users, you need to download and start ChromeDriver, ensuring its version matches your browser version.

Copy

driver = webdriver.Chrome()

During the debugging phase, we do not recommend starting with headless mode, as a visual browser can significantly reduce the difficulty of locating elements.

3. Install Pandas

To keep the scraped data organized, we need Pandas for data cleaning and exporting

Copy

pip install pandas

Scraping YouTube Video Data

Our goal is clear: to access the video page of a YouTube channel, sort by "Most popular," and extract the title, views, and publish time of each video.

Step 1: Initialize the Driver and Target Location

First, we cannot simply scrape from the channel homepage. To ensure data accuracy, we will manually filter "Most popular" in the browser and then copy the URL that includes the sorting parameter.

Copy

from selenium import webdriver
import pandas as pd
import time

# Initialize Chrome driver
driver = webdriver.Chrome()

# Target URL: Pre-sorted YouTube video page
url = "https://www.youtube.com/@TargetChannel/videos?view=0&sort=p&flow=grid"
driver.get(url)

# Give the page some loading time; dynamic web pages need to wait for JS rendering
time.sleep(5)

Step 2: Analyze the DOM Structure

When scraping a dynamic web page, the most critical step is "Inspect." Right-click on the video title, and you will find that all video information is wrapped in a specific container.

What we want to do is not to directly scrape all the titles, but to first grab these "containers" and then recursively look for child elements inside each container.

● Parent container class name: style-scope ytd-video-renderer

● Title element: Usually located within the tag with id="video-title".

● Metadata (views and time): Located within id="metadata-line" and its child span tags.

Step 3: Core Scraping Logic

This is the most critical step of the entire task. We need to locate all video containers and then iterate through them. Here, we want to emphasize the importance of "Relative XPath."

Copy

# Get all video containers
videos = driver.find_elements_by_class_name('style-scope ytd-video-renderer')

video_list = []

for video in videos:
    try:
        # Use relative XPath to extract the title
        title = video.find_element_by_xpath('.//a[@id="video-title"]').text
        
        # Extract views and posting time
        # Usually, both are in the same meta section
        meta_data = video.find_element_by_xpath('.//div[@id="metadata-line"]').text
        meta_lines = meta_data.split('\n')
        
        views = meta_lines[0] if len(meta_lines) > 0 else "N/A"
        posted_time = meta_lines[1] if len(meta_lines) > 1 else "N/A"

        video_data = {
            'title': title,
            'views': views,
            'posted': posted_time
        }
        video_list.append(video_data)
    except Exception as e:
        print(f"Error extracting: {e}")
        continue

Why Should We Use Relative XPath?

If you do not add a dot (.) in the loop, Selenium will by default start searching from the entire page's DOM, resulting in you always getting the data for the first video on the page. This is the most common pitfall for beginners in web scraping dynamic content.

Here are a few key details:

• .//: Limit the search scope

• .text: Retrieve the visible text for the user

• Use list indexing instead of repeating XPath to improve performance and stability

This step effectively addresses the most common "data misalignment" issue in scraping dynamic websites.

Step 4: Data Processing and Output

After scraping the raw data, we cannot let it scatter in memory. Using Pandas, we can quickly convert it into a structured DataFrame, which is not only visually appealing but also convenient for subsequent analysis.

Copy

# Convert the list to a DataFrame
df = pd.DataFrame(video_list)

# Simple cleaning logic: remove the "views" string from views count, keeping only the numbers (optional)
df['views_count'] = df['views'].str.replace(' views', '')

# Output display
print("Summary of the captured video data:")
print(df.head())

# Export to CSV file
df.to_csv('youtube_data.csv', index=False, encoding='utf-8-sig')

# Close the driver
driver.quit()

Through Pandas, we can quickly check the integrity of the data. For example, if the number of videos scraped is less than expected, we may need to add WebDriverWait to handle the delays in dynamic loading.

Integrating Thordata Scraping Solution

When we transition from local scripts to large-scale, production-level scraping tasks, we often encounter challenges such as IP blocking, CAPTCHA, and fingerprint recognition. At this point, we need a more specialized web scraping solution.

Thordata's proxy infrastructure and web data scraping solutions provide comprehensive support, making complex scraping tasks simpler:

• High-quality Residential Proxies: Provide over 100M residential IP addresses from real users, significantly reducing the risk of being identified as a bot by the target website.

• Static ISP Proxies: Combine the high-speed response of data centers with the high success rate of residential IPs, suitable for sessions that require long-term stability online.

• Mobile Proxies: Access through real mobile networks (4G/5G), capable of penetrating even the strictest anti-scraping strategies.

• Scraping Browser: Built-in automatic rendering and fingerprint masking features, so you don’t have to worry about complex browser configurations.

• Ready-To-Use Datasets: If you don't want to write code, you can directly obtain cleaned structured data.

To make your Selenium scripts more powerful, we can easily integrate them into Thordata's scraping browser.

Copy

from selenium import webdriver

# Thordata remote connection address for the scraping browser
# Include your API credentials and regional configuration
THORDATA_REMOTE_URL = "http://USER:PASS@proxy.thordata.com:PORT"
options = webdriver.ChromeOptions()

# Connect to Thordata's scraping environment using the remote WebDriver
driver = webdriver.Remote(
    command_executor=THORDATA_REMOTE_URL,
    options=options
)

driver.get("https://www.youtube.com/...")

# The subsequent scraping logic is the same as local
print(driver.title)
driver.quit()

Sign up for Thordata now - Free Trial of Web Scraping Solutions!

Compliance and Ethics: The "Unwritten Rules" of Scraping

After mastering the powerful skill of scraping dynamic websites, we must talk about the "rules."

Web scraping is not a lawless territory; when we engage in data collection activities, we must adhere to the following principles:

1. Follow Robots.txt: Before scraping, check what content the target website allows to be scraped.

2. Control the scraping frequency: Do not send thousands of requests in a short period, as that is akin to a DDoS attack. Reasonable delays (Time Sleep) are a basic respect for the server.

3. Protect privacy data: If user personal information is involved, be sure to comply with GDPR or relevant data protection laws.

4. Commercial use declaration: If the data you scrape is for commercial profit, ensure that this does not violate the target website's Terms of Service (ToS).

Remember, a sustainable scraping project must be built on a foundation of compliance.

Advanced Techniques: Moving Toward Automation

Once you have mastered dynamic website scraping using Python, the next challenge is how to run it stably in a large-scale environment.

• Headless Mode: In a Linux server or Docker container, we do not need to display the browser interface. Enabling headless mode can significantly reduce memory usage.

• Explicit Waits: Instead of rigidly using time.sleep(5), it is better to use Selenium's WebDriverWait to wait for specific elements to appear. This balances speed and stability.

• Exception Handling: Network fluctuations are the norm. Adding a robust Try-Except block in your loop can ensure that your script does not crash due to a minor loading failure.

Conclusion

Through the discussion in this article, we have not only mastered the powerful skill of scraping dynamic websites but also hands-on implemented a complete process for extracting dynamic data from YouTube. Selenium equips users with the ability to simulate human behavior, while professional scraping tools like Thordata enhance the stability and efficiency of the scraper. Scraping dynamic websites is not just about "writing code," but rather a systematic, strategic, and sustainable approach to data acquisition.

We hope the information provided is helpful. However, if you have any further questions, feel free to contact us at support@thordata.com or via online chat.

 

Get started for free

Sign up with Google