web-scraping.dev

A realistic e-commerce testing platform for web scraping developers

Practice web scraping with 19 realistic scenarios covering pagination, authentication, GraphQL APIs, CSRF protection, and more. Safe, legal, and designed for learning.

Realistic Scenarios Challenging CI Testing Suite
Explore Scenarios Try Demo

Used By

ScrapFly Documentation Web Scraping Academy Blog & Tutorials

Scraping Scenarios 57 Scenarios

Real-world web patterns you'll encounter in production scraping projects. Each scenario is tagged with difficulty level and includes working code examples.

Scenario Path Level Description
Static Paging https://web-scraping.dev/products Beginner HTML-based server-side item paging where each page is its own URL.
Endless Scroll Paging https://web-scraping.dev/testimonials Intermediate Dynamic client side paging where new items appear as user scrolls.
Secret API Token https://web-scraping.dev/testimonials Intermediate X-Secret-Token used to lock access to hidden APIs.
Endless Button Paging https://web-scraping.dev/reviews Intermediate Dynamic paging where new items appear on Load More click.
GraphQL Background Requests https://web-scraping.dev/reviews Advanced Data loaded via JavaScript through a backend GraphQL API.
Forced New Tab Links https://web-scraping.dev/reviews Beginner Links forced to open in new tabs via different techniques.
Product HTML Markup https://web-scraping.dev/product/1 Beginner Basic e-commerce product structure and CSS class based markup.
Hidden Web Data https://web-scraping.dev/product/1 Intermediate Review data hidden in HTML as JSON, loaded on page load.
Local Storage https://web-scraping.dev/product/1 Intermediate Cart system powered by localStorage for client-side state.
CSRF Token Locks https://web-scraping.dev/product/1 Advanced Load More reviews uses X-CSRF-Token header to block cross-site access.
Cookies Based Login https://web-scraping.dev/login Intermediate User authentication based on form request and cookies.
Iframe Login https://web-scraping.dev/login/iframe Intermediate Login form embedded in an iframe (SSO widget pattern).
PDF Downloads https://web-scraping.dev/login Intermediate Link and JS based file download triggers.
Cookie Popup https://web-scraping.dev/login?cookies= Beginner Cookie info modal popup that blocks the entire screen.
Example Block Page https://web-scraping.dev/blocked Beginner Valid 200-status response that redirects to block notification.
Blocking Redirect (Invalid Referer) https://web-scraping.dev/credentials Advanced Page requires valid Referer header, otherwise redirects to blocked.
Persistent Cookie Blocking https://web-scraping.dev/blocked?persist= Advanced Using cookies to mark blocked clients for persistent blocking.
Form File Attachment Download https://web-scraping.dev/file-download Intermediate Form submission triggers file download with Content-Disposition header.
AI Content Obfuscation https://web-scraping.dev/ai-content-obfuscation Intermediate Extract clean text from AI-obfuscated content using invisible Unicode.
Challenge + File Download https://web-scraping.dev/challenge-download Advanced Challenge bypass + file download with 403 status.
Bad Encoding https://web-scraping.dev/bad-encoding Intermediate Pages with mismatched Content-Type charset headers.
Antibot Challenge https://web-scraping.dev/antibot/easy Beginner Simple antibot protection that blocks direct access.
Robots.txt Compliance Trap /robots-disallowed Crawler Disallowed URL that returns 200. Conforming crawlers must not fetch.
WebMCP Tools /mcp-tools Intermediate Page with registered MCP tools. Test native browser MCP with AI agents via Cloud Browser.
Classify API: Blocked Fixture /classify-test/blocked Beginner Returns a 403 Cloudflare-style challenge. Feed the response to POST /classify to exercise the block-detection pipeline.
Classify API: Clean Fixture /classify-test/clean Beginner Returns a legitimate 200 JSON response. Feed it to POST /classify to verify the endpoint correctly reports blocked=false.
Long Page Screenshot https://web-scraping.dev/long-page Intermediate A page far taller than one screen. Full-page capture has to take it in sections and join them; numbered bands and an END OF PAGE marker show when a section repeats or goes missing.
LocalStorage Merge & Bomb https://web-scraping.dev/storage-test Advanced Test scrape-engine session storage merge across rescrapes. ?seed=1 plants keys, ?bomb=1 writes a 1MB value (rejected by 500KB cap), ?delete=k1 verifies removal propagates.
Cross-Site Iframe Subresource /oopif-subresource Advanced A cross-site iframe pulls a 400KB immutable script. The frame runs in its own renderer, so the request belongs to the frame's CDP session, not the page's — interception enabled only on the page misses it entirely and the bytes are re-downloaded every scrape.
Crawler Test Report Endpoint /crawler-test-report Crawler Central JSON assertion endpoint. POST /reset to clear between runs.
Rate Limit & Politeness /rate-limited Crawler 429 with Retry-After: 5 after 3 req/10s.
Crawl Delay Enforcement /slow-section/page/1 Crawler robots.txt Crawl-delay: 2 for /slow-section/.
Meta Robots Directives /meta-noindex Crawler Pages with meta noindex/nofollow.
X-Robots-Tag Header /header-noindex Crawler Responses with X-Robots-Tag noindex/nofollow.
Canonical URL Handling /canonical-tracking?utm_source=foo&session=bar Crawler Tracking params with rel=canonical.
Session ID Vortex (Trap) /session-vortex Crawler Every page produces 10 new sid-parameterized links.
Infinite Calendar (Trap) /calendar/2024/01 Crawler Calendar with prev/next links across 2000-2100.
Redirect Chain & Loops /redirect-chain/1 Crawler Chain of 10 redirects + loop detection.
Fragment & URL Normalization /fragment-collapse Crawler Fragment-only and normalization variants.
Content Deduplication /dup-a Crawler Identical HTML with canonical. Tests dedup.
JS-only Link Discovery /js-links Crawler Anchors injected by JS after DOMContentLoaded.
JSON-LD Link Discovery /linked-data Crawler Link only present inside a JSON-LD script tag.
data-href Link Discovery /data-href Crawler Link attribute on a non-anchor element.
HTTP Link Header Discovery /header-link Crawler Link: rel=next HTTP header points to target.
HTML base Tag /base-tag Crawler Relative links resolved against base href.
Huge Response Body /huge-page Crawler Streams ~50MB of HTML.
Slow Drip Response /slow-drip Crawler 1 byte/sec for 60s. Tests timeout behavior.
Wrong Content-Length /wrong-content-length Crawler Header claims 100 bytes, body is 1000.
External Redirect /redirect-external Crawler 302 to example.com.
Mixed HTTP/HTTPS Content /mixed-content Crawler Same URL under http and https.
Sitemap Index /sitemap-index.xml Crawler 50 child sitemaps x 100 URLs each.
Gzipped Sitemap /sitemap.xml.gz Crawler sitemap.xml as application/gzip.
Basic Auth Wall /private/secret Crawler 401 + WWW-Authenticate: Basic.
Cookie Consent Flow /needs-consent Crawler Multi-step cookie consent redirect.
Vary: Origin CORS Chunk /vary-origin Cache Cross-origin chunk whose body names the requesting Origin, declared with Vary: Origin. A cache keyed on the URL alone replays one origin copy and Chromium blocks the script. ?chunk= serves the chunk from another host.
ACAO Without Vary /acao-no-vary Cache Same origin-specific chunk with NO Vary header, the common CDN shape. A concrete Access-Control-Allow-Origin has to make the entry origin-specific on its own.
Byte-Range Fragments /range-page Cache Drives real Range requests against /range-asset (4096 bytes, strong ETag) and reports the first byte of each slice. /range-asset?weak=1 downgrades the validator, which a cache must refuse to store fragments for.
CORS Byte-Range /cors-range-page Cache crossorigin Range fetch of /cors-range-asset, whose first 64 bytes name the origin the slice was built for. Reveals a cache that stores range fragments without keying them by origin.