web-scraping.dev

HTML-based server-side item paging where each page is it's own URL.

Dynamic client side paging where new items appear as user scrolls.

The testimonial paging shows how X-Secret-Token is used to lock access to hidden APIs.

Dynamic client side paging where new items appear as presses Load More button.

Data loaded using javascript through a backend graphql API.

The review policy and code of conduct links use different techniques to force opening links in a new tab.

Basic e-commerce product structure and css class based markup.

Product review data is hidden in HTML as json, then loaded to HTML on page load.

The cart system is powered by local storage to demo client-side website state store (alternative to cookies).

The Load More reviews action uses a X-CSRF-Token header to block cross-site access.

User authentication based on form request and cookies.

Login form embedded in an iframe - a common pattern for SSO widgets and third-party auth that requires frame-aware scraping.

The login page features link and js based file download triggers.

Cookie info modal pop up that blocks the entire screen

Valid 200-status response that redirects to block notification.

The credentials page is only accessed with valid Referer otherwise redirects to blocked. So this page can only be accessed by clicking link on /login.

Using cookies to mark blocked clients for persistent blocking.

Form submission that triggers a file download with Content-Disposition attachment header in a new tab.

Extract clean cipher text from AI-obfuscated content using invisible Unicode characters that confuse LLMs and AI scrapers.

Simulates challenge bypass (like Cloudflare Turnstile) that triggers a file download with 403 status. Tests ASP shield bypass + attachment handling.

Pages with mismatched Content-Type charset headers: Cyrillic, German, Chinese content with wrong encoding declarations.

Simple antibot_easy protection that blocks direct access. Can you figure out how to get past it?

Central JSON assertion endpoint aggregating all crawler trap hits. POST /crawler-test-report/reset to clear between runs.

Returns 429 with Retry-After: 5 after 3 req/10s. Records rate_limit_ignored if retried early.

robots.txt declares Crawl-delay: 2 for /slow-section/. Hits closer than 2s record crawl_delay_ignored.

Pages with meta noindex/nofollow and rel=nofollow links. Targets record violations when fetched.

Responses carrying X-Robots-Tag noindex/nofollow. Targets record violations.

Page with tracking params and rel=canonical. Report reveals canonical vs tracking URL storage.

Every page produces 10 new sid-parameterized links. Dedup-aware crawlers should stop quickly.

Calendar with prev/next month links across 2000-2100. Report shows max depth reached.

Chain of 10 redirects and /loop-a <-> /loop-b loop. Records redirect_chain_depth and redirect_loop.

Fragment-only and normalization variants. Conforming crawlers dedupe to a single fetch.

/dup-a and /dup-b serve identical HTML with canonical to /dup-a. Tests dedup via canonical.

Anchors injected by JS after DOMContentLoaded. Requires JS-capable crawler to discover the target.

Link only present inside a JSON-LD script tag. Tests schema.org-aware discovery.

Link attribute on a non-anchor element. Tests heuristic link discovery.

Link: rel=next HTTP header points to the target. Tests header-based discovery.

Relative links resolved against . Target records base_tag_respected.

Streams ~50MB of HTML. Reveals if the crawler sends Range headers or truncates.

StreamingResponse 1 byte per second for 60s. Tests connection timeout behavior.

Header claims 100 bytes but body is 1000. Tests HTTP resilience.

302 to example.com. Verifies follow_external handling.

Same URL linked under http and https. Tests scheme-aware dedup.

50 child sitemaps x 100 URLs each. One entry is a dead 404. Tests sitemap index traversal and resilience.

sitemap.xml served as application/gzip. Tests transparent decompression.

401 + WWW-Authenticate: Basic. Records authenticated boolean on each hit.

Multi-step cookie consent redirect. Records consent_flow_completed only when the full flow runs.