AutoScraper Python: Smart Scraping and Proxy Guide

Imagine scraping a website without ever pressing F12 to inspect the source code. Imagine just telling Python: “I want the data that looks like ‘iPhone 15′” and having the script figure out the CSS selectors automatically.

This is the promise of AutoScraper, a “Smart, Automatic, Fast and Lightweight Web Scraper for Python.” It uses a form of fuzzy matching to learn scraping rules from the examples you provide.

While many blogs cover the “Hello World” of AutoScraper, few show how to use it in the real world—where websites ban IPs and structures get complex. In this guide, I’ll take you from the basics to an advanced setup with proxy integration.

1. The “Magic” of Learning by Example

First, install the library. It is lightweight and depends on requests and bs4.

pip install autoscraper

Let’s say we want to scrape book titles from a test site. Instead of looking for div.product_pod h3 a, we just pick a title we see on the page, like “A Light in the Attic”, and give it to AutoScraper.

from autoscraper import AutoScraper

url = 'https://books.toscrape.com/'

# We want to scrape titles and prices. We give one example of each.
wanted_list = ["A Light in the Attic", "£51.77"]

scraper = AutoScraper()

# The build() function learns the rules
result = scraper.build(url, wanted_list)

print(result)
# Output: ['A Light in the Attic', 'Tipping the Velvet', ..., '£51.77', '£53.74', ...]

2. Grouping Data & Saving Models

The output above is a flat list. In a real project, you want a dictionary where titles are linked to prices. We can ask AutoScraper to group the rules and then save the model for later use.

# Get the rules learned
rules = scraper.get_result_similar(url, grouped=True)

print(rules.keys()) 
# Output: dict_keys(['rule_1ab2', 'rule_8x9y'])

# Alias the rules and save the model
scraper.set_rule_aliases({'rule_1ab2': 'Title', 'rule_8x9y': 'Price'})
scraper.keep_rules(['rule_1ab2', 'rule_8x9y'])
scraper.save('books_model')

3. The Missing Piece: Proxy Integration

This is where most AutoScraper tutorials fail. If you try to run your saved model on Amazon or Google, you will be blocked immediately. AutoScraper uses the requests library internally, which means we can pass arguments to it via request_args.

from autoscraper import AutoScraper

# Load the trained model
scraper = AutoScraper()
scraper.load('books_model')

# Thordata Proxy Configuration (Username:Password)
proxies = {
    "http": "http://USER:PASS@gate.thordata.com:12345",
    "https": "http://USER:PASS@gate.thordata.com:12345",
}

# Scrape a new page using the proxy
target_url = 'https://books.toscrape.com/catalogue/page-2.html'

results = scraper.get_result_similar(
    target_url, 
    group_by_alias=True,
    request_args={'proxies': proxies, 'timeout': 10}
)

print(results['Title'][0])

4. When to Use (and Avoid) AutoScraper

As a data engineer, picking the right tool is half the battle. Here is my verdict on when AutoScraper shines and when it falls short:

Conclusion

AutoScraper is a brilliant tool for “lazy” scraping. It lowers the barrier to entry significantly. However, for enterprise-grade data extraction, it must be paired with robust infrastructure. By adding Thordata Residential Proxies to the mix, you turn this lightweight tool into a capable scraper for moderate workloads.