Skip to content

AI Compass / Knowledge cards

Collecting web data

settle the access path first, then build

Checked Automation Solo business

A scraper is a program that fetches web pages automatically and extracts data from them. Claude Code can build such a program in a short time today. The difficulty usually lies not in the code but in where the requests come from and whether they are allowed.

What it is for

Market and price research without an expensive subscription, without running into legal problems.

Steps

  1. Look for official routes first: APIs, RSS feeds or data exports from the target site.
  2. Read the site’s robots.txt. In it, the website specifies what programs may fetch. Read the terms of use as well.
  3. Fetch only public pages, slowly and with pauses between requests.
  4. Expect blocks: some sites block requests from data centers. In the measurement below, a scraper got through on only 33 of 100 pages that way.
  5. If a page is blocked, don’t circumvent the block. Use another route or skip the page.
  6. Don’t store personal data you don’t need.

What the example shows

Creator Magic had Claude Code build a scraper that respects robots.txt. From a data center it reached 33 of 100 pages. Via residential proxies, meaning detours through private internet connections, it was over 90%, at a total cost of around $2. That is a single measurement by the author on 100 pages, not verified and not easily repeatable.

Respecting robots.txt while circumventing blocks via proxies is contradictory. That is why this variant is not listed here as a step.

Alternatives

Sources

Status

Checked 2 Oct 2026 · Maturity new · not legal advice

Want to try this with your team? That’s what the AI Lab is for. See the AI Lab