---
name: python-web-scraper-builder
description: "Boilerplate for a polite Python scraper that pulls the fields you name from a site, handles pagination, and writes the results where you want them, with comments on running it without getting blocked. Use when data such as product listings or article metadata needs pulling from a website into CSV, JSON, or a database with a Python script. Do not use for sites that offer an official API, or scraping that would breach a site's terms or collect personal data. Related topics: coding, python, data."
---

# Python Web Scraper Builder

Boilerplate for a polite Python scraper that pulls the fields you name from a site, handles pagination, and writes the results where you want them, with comments on running it without getting blocked.

## When to use

Use this skill when data such as product listings or article metadata needs pulling from a website into CSV, JSON, or a database with a Python script. Do not use it for sites that offer an official API, or scraping that would breach a site's terms or collect personal data.

## Workflow

1. Ask the user for each input listed below in a single message, skipping any they have already given. Where options are listed, offer them, but accept other answers too.
2. Choose static parsing or a browser driver based on whether the target content is rendered by JavaScript, and state the reason in a comment.
3. Write the fetch, parse, and write stages as separate functions with the selectors grouped at the top of the file.
4. Add polite defaults: a request delay, a descriptive User-Agent, a robots.txt check, and a stop after repeated HTTP errors.
5. Implement pagination and login only when selected, with credentials read from environment variables.
6. Finish with install and run instructions and a two-sentence note on responsible use.

## Inputs

- **Target URL** (free text, e.g. "https://example.com/products")
- **Data to extract** (longer free text)
- **Library** (choose one: BeautifulSoup + Requests, Scrapy, Selenium, Playwright)
- **Output** (choose one: Print to console, Save to CSV, Save to JSON, Insert into Database)
- **Handle pagination** (choose one: Yes, No)
- **Extras** (choose any: Use proxies, Randomize User-Agent, Add delay between requests, Handle login)

## Output

A single python code fence with the commented script, followed by the pip install line, the run command, and a short note on rate limits and terms of service.

## Prompt template

Fill the inputs into this template and follow it as the full instruction.

### Python Web Scraper

Build a Python scraper for one site.

- Target URL: {{Target URL}}
- Data to extract: {{Data to extract}}
- Library: {{Library}}
- Output: {{Output}}
- Handle pagination: {{Handle pagination}}
- Extras: {{Extras}}

Pick the extraction approach that matches the site and library: CSS selectors for static HTML, a browser driver only when the content is rendered by JavaScript. Keep the selectors in one place near the top of the file so they are easy to update when the markup changes. Separate fetching from parsing from writing, so each part can be tested on its own. Add a delay between requests and a descriptive User-Agent by default, respect robots.txt, and stop on repeated HTTP errors instead of hammering the server. When pagination is on, follow the next link until it is missing and log the page count. When login is on, read credentials from environment variables.

Return the complete script in one code fence labeled python with comments on each section, then a short block: the pip packages to install, the command to run it, and two sentences on running it responsibly (request rate, terms of service, personal data).

## Pitfalls

- Using Selenium or Playwright for a static page, which makes the scraper ten times slower and harder to run on a server.
- Scattering selectors through the parsing code, so one markup change means edits in five places.
- Shipping without a delay or error cap, which gets the IP blocked on the first full run.
