web-scraping.dev

web-scraping.dev is a mock website for testing and learning about web scraping. It covers popular web patterns encountered in web scraping like:

  • Item pagination
  • Item pagination in SPA's (e.g. load more button)
  • Endless item pagination
  • Data loading from embeded JSON
  • Data loading through XHR (background requests)
  • E-commerce product data structures
  • Local Storage for cart tracking

This platform is used in ScrapFly web scraping API documentation and blog tutorials , so check them out if you'd like to learn more!

Changelog

v1.2.0
  • Change header requirement for /api/reviews to require only x-csrf-token header (secret-csrf-token-123)
  • Change header requirement for /api/testimonials to require only referer header (https://web-scraping.dev/testimonials)
v1.1.0
  • Add cookie popup modal to /login
  • Add cart system: see cart preview button at the top and the /cart endpoint; enable add to cart button on products. Carts are purely JS and are used to demo Local Storage
  • Add header requirements to /api/reviews for Referer and X-Csrf-Token to demo header locking
  • Add multiple product request api through post to /api/products with multiple id values, e.g. {"id": [1,2,3,4]}
  • Improve styling, especially for mobile
  • Improve openapi docs with examples, default values and more info (/docs)