Python Web Scraper Builder skill for Claude and Codex
Boilerplate for a polite Python scraper that pulls the fields you name from a site, handles pagination, and writes the results where you want them, with…
Install it once and Claude or Codex runs it for you. When your request matches, your AI loads the skill, asks for anything it is missing, and works through the template itself. Prefer to fill it in by hand? See the full prompt template.
Download the skill (.zip) · View SKILL.md
When it is used
Use when data such as product listings or article metadata needs pulling from a website into CSV, JSON, or a database with a Python script. Do not use for sites that offer an official API, or scraping that would breach a site's terms or collect personal data. Related topics: coding, python, data.
What it will ask you for
- Target URL (free text, e.g. "https://example.com/products")
- Data to extract (longer free text)
- Library (choose one: BeautifulSoup + Requests, Scrapy, Selenium, Playwright)
- Output (choose one: Print to console, Save to CSV, Save to JSON, Insert into Database)
- Handle pagination (choose one: Yes, No)
- Extras (choose any: Use proxies, Randomize User-Agent, Add delay between requests, Handle login)
How it works
- Choose static parsing or a browser driver based on whether the target content is rendered by JavaScript, and state the reason in a comment.
- Write the fetch, parse, and write stages as separate functions with the selectors grouped at the top of the file.
- Add polite defaults: a request delay, a descriptive User-Agent, a robots.txt check, and a stop after repeated HTTP errors.
- Implement pagination and login only when selected, with credentials read from environment variables.
- Finish with install and run instructions and a two-sentence note on responsible use.
What you get
A single python code fence with the commented script, followed by the pip install line, the run command, and a short note on rate limits and terms of service.
What it watches out for
- Using Selenium or Playwright for a static page, which makes the scraper ten times slower and harder to run on a server.
- Scattering selectors through the parsing code, so one markup change means edits in five places.
- Shipping without a delay or error cap, which gets the IP blocked on the first full run.
How to install
Claude.ai and the Claude desktop app
- Download the skill .zip.
- Open Settings, then Capabilities, and find Skills.
- Choose Upload skill and select the .zip.
- Ask Claude for help with the task. It loads the skill when your request matches.
Claude Code
- Download the skill .zip.
- Unzip it into ~/.claude/skills/ (or .claude/skills/ inside a project).
- Start a new session. Claude Code loads the skill when your request matches.
Codex
- Download the skill .zip.
- Unzip it into ~/.agents/skills/ (or .agents/skills/ inside a project).
- Codex picks it up on its own. Type $ and the skill name to use it, or just ask for the task.
Cursor
- Download the Cursor rule (.mdc).
- Save it in .cursor/rules/ inside your project.
- Cursor's agent applies the rule when your request matches its description.
AGENTS.md (Codex, Cursor, and other coding agents)
- Copy the AGENTS.md section.
- Paste it into AGENTS.md at the root of your repository, creating the file if needed.
- Coding agents that read AGENTS.md follow it when your request fits.