Best web scraping test sites

A scraper pointed at a load more button finishes in under a second, exits cleanly and writes six records. The category holds several hundred. Nothing errored, no request failed, and the run is marked complete.

That failure is the reason to use a practice site, and it is the thing most lists of them never mention. You get a set of URLs, a note that one holds books and another holds quotes, and no indication of which production problem each one reproduces.

These sites are not interchangeable. A static catalogue of a thousand products tests whether your selectors are correct and whether your crawler reaches the last page. It cannot tell you whether your scraper distinguishes a page that loaded from a page that contains data. The list below is ordered by diagnostic value rather than popularity, and each entry states what the site tests, what it does not, and which production failure it stands in for.

The ranking below rewards sites you can learn something from when they fail, rather than sites that are simply well known. The Web Scraper test sites come first because they expose a single catalogue through four different access mechanisms, which makes them the only entry here where you can change the pagination method while holding the data constant.

Site	Primary use	Known record count	Needs a browser
1. webscraper.io/test-sites	Pagination and session mechanisms	Yes, 17 pages on the pagination variant	Depends on the variant
2. books.toscrape.com	Selectors, attributes, count assertions	Yes, 1,000 books over 50 pages	No
3. quotes.toscrape.com	JavaScript rendering and login	No	Depends on the variant
4. scrapethissite.com	Forms, frames, headers, cookies	Partial	On the AJAX lesson only
5. httpbin.org	Status codes, redirects, delays	Not applicable	No
6. scrapeme.live	Production-style e-commerce markup	Roughly 755 products over 48 pages	No
7. UI automation sandboxes	Dynamic loading, dynamic IDs, shadow DOM	No	For most examples

Every site listed was checked in August 2026. Practice sites go offline without notice, and two of the seven are third-party projects with nobody obliged to keep them running. Where that applies, the entry says so.

How this list is ordered

Four criteria, applied in this order:

1. Whether the site lets you change one variable at a time
2. How many distinct scraping behaviours it exercises
3. Whether it has a known record count you can assert against
4. Whether it is stable and maintained

The first criterion carries the most weight because it decides whether a failed run tells you anything. If a site changes its markup, its data and its pagination method all at once, a failure has several plausible causes and you are back to guessing. A site that exercises more behaviours than any other still ranks below one that lets you isolate a single cause, which is why the order below is not simply a count of features.

1. Web Scraper test sites

Best for: isolating pagination and page-state behaviour.

The Web Scraper test sites present four variants of the same car catalogue, differing only in how items are reached: standard pagination links, a load more button, infinite scroll, and a catalogue behind a login.

That structure is what puts them first. Everywhere else on this list, changing the pagination mechanism also changes the site, the markup and the data. Here the catalogue is constant and only the access mechanism varies, so a failure points at the mechanism rather than at your selectors.

The four variants

• /test-sites/pagination serves 17 pages of ?page= links across brand categories and subcategories. Each item carries name, description, year, country of origin, mileage, price and an availability status of Available, Reserved or Sold. That last field is a closed set of three values, which makes it directly validatable: if some records come back with an empty availability, you have a parsing bug rather than a target problem.
• /test-sites/load-more returns six items in the initial HTML behind a load more button, with no link-based fallback.
• /test-sites/scroll returns the same six and appends more as the page scrolls.
• /test-sites/website-state-setup-login gates the catalogue behind a login, with credentials published on the page.

The load more and scroll variants produce the most instructive failure on this list. That failure is worth seeing rather than describing. Below is the complete output of a plain HTTP request to /test-sites/scroll, every record it returns:

• Mercedes-Benz W123 280E 1955
• BMW E24 635CSi 1954
• Jaguar XJ6 1983
• Nissan 300ZX Z31 1966
• Ferrari F40 1984
• Mercedes-Benz W123 280E 1972

A browser keeps loading past that point. The HTTP client does not, and it exits successfully. The request succeeded, the page loaded, the scraper completed, records were extracted, and the dataset is still wrong.

The login variant deserves a separate look, because an unauthenticated request returns HTTP 200 with a login form rather than a 401. The status line reports success while the body contains no data, which is structurally identical to a consent screen or a soft block. In Web Scraper, this page exists to demonstrate Website State Setup, which performs the login before extraction starts so the session already exists when the job runs.

What it does not cover: no anti-bot layer, no rate limiting, no markup drift. It tests your configuration, not your resilience.

2. books.toscrape.com

Best for: first selectors, attribute extraction and record count assertions.

The default recommendation, and for a defensible reason. A thousand books, twenty per page, fifty pages, a detail page each, and a stated record count that turns "the job finished" into something you can check.

The instructive field is the star rating, because it is not text. It appears as a second class on the rating element, as in <p class="star-rating Three">, so extracting it means reading a class attribute and mapping a word to a number. That is the first point at which a scraper configured to collect visible text quietly returns nulls, which makes it a useful early lesson in choosing selectors that target the right thing.

The site states plainly that it is a demo and that its prices and ratings were randomly assigned, so do not use it to sanity-check parsing logic that depends on plausible values.

What it does not cover: anything dynamic. Everything is server-rendered.

3. quotes.toscrape.com

Best for: rendering decisions, wait strategies, login and CSRF.

The broadest set of behaviours on one domain. The same quote data is served through eight variants, which makes it the closest thing available to a rendering test suite.

Variant	What it tests
/	Microdata and standard pagination
/js	Content written into the DOM by script, absent from the initial HTML
/js-delayed?delay=10000	Whether your wait strategy targets data presence or page load
/scroll	Infinite scrolling pagination
/login	CSRF token extraction and cookie persistence, any credentials accepted
/tableful	Table-based layout with broken markup
/search.aspx	An ASP.NET AJAX filter form with ViewState parameters
/random	A single record, useful for smoke tests

The /js and /js-delayed pair is the most valuable part. The first establishes a diagnostic worth keeping permanently in your workflow, and the second separates two things people routinely conflate.

On /js-delayed, a scraper that waits for the load event and reads the DOM immediately returns empty records and reports success while doing it. That is the difference between the page having loaded and the data being present. How JavaScript-rendered content affects web scraping covers what committing to rendering costs once you are running at volume.

/search.aspx is niche and exactly right if your real target is an ASP.NET application, because ViewState means you cannot construct the next request URL by hand.

What it does not cover: no known record count, so you cannot build a count assertion against it.

4. scrapethissite.com

Best for: search forms, frames, and the header and cookie layer.

Five structured lessons, each isolating a different obstacle:

• Countries of the world on a single page, minimal markup, good for a first selector and nothing more
• Hockey teams puts NHL statistics since 1990 behind a search form with pagination, so query parameters and result pages interact rather than being separate exercises
• Oscar winning films adds data asynchronously after render, which is the right place to practise the cheaper alternative to a browser: open the network tab, find the request returning the data, and call that endpoint directly. Web scraping vs API sets out when that is the better route
• Frames and iFrames, where content sits in a separate document and a selector evaluated against the parent will not find it however correct it looks
• Advanced topics combines header spoofing, session cookies, CSRF tokens and common network errors

The advanced lesson is the one most people skip and most people need. It is the only entry on this list that puts request-level obstacles and parsing in the same exercise.

What it does not cover: scale. Every lesson is small by design.

5. httpbin.org and httpbingo.org

Best for: retry logic, timeouts, redirects and compression.

These are not parsing exercises. They test the layer underneath, which is where most production reliability problems actually live.

• /status/:code returns any status on demand. Point your retry logic at 429, 500 and 503 and confirm it backs off rather than retrying immediately
• /delay/:n and /drip test timeout handling and slow or streamed responses
• /redirect/:n, /absolute-redirect/:n and /redirect-to test redirect following and redirect limits
• /cookies/set and /cookies/delete test cookie jar behaviour across requests
• /basic-auth, /digest-auth and /bearer cover the common auth schemes
• The gzip, deflate and brotli endpoints confirm your client decompresses what it claims to accept

Both run locally in Docker, which is worth doing. The public instances are shared and rate limited, so a retry test against them measures someone else's traffic as much as your own backoff implementation.

What it does not cover: no HTML worth parsing, and the status codes are returned on demand rather than earned. This teaches you how your client reacts, not when a real target will make it react.

6. scrapeme.live

Best for: realistic e-commerce markup.

A working WooCommerce store of roughly 755 products across 48 pages, with sorting, product pages, prices and pagination. The value is that it was not written to be scraped. It is a real theme, so the class names, nesting depth and template noise behave the way a production storefront does, including the parts that make selectors awkward.

Use it as the step between purpose-built sandboxes and a real target. If your sitemap handles this cleanly, the problems that remain are access problems rather than parsing problems.

One caveat: it is a third-party demo store with no maintenance commitment behind it. Treat it as useful while it lasts rather than as a fixture in an automated test suite.

What it does not cover: still no anti-bot layer, and a WooCommerce theme is only one of the platforms you will meet.

7. UI automation sandboxes

Best for: dynamic loading, auth schemes and DOM edge cases, each in isolation.

Two sites built for UI test automation rather than scraping. A good half of what they offer (alerts, drag and drop, file upload) is irrelevant here, but the DOM problems are identical to the ones that break scrapers, and each is isolated on its own page with nothing else happening.

the-internet.herokuapp.com covers dynamic loading, infinite scroll, shifting content, status codes, and basic and digest authentication. Its infinite scroll appends text indefinitely, which makes it useful for a reason the others are not: it tests whether your scraper has a stopping condition rather than an end condition. A crawler that waits for the last page will never stop.

uitestingplayground.com covers dynamic IDs, AJAX-loaded elements, client-side computation delays, elements hidden behind z-order, and shadow DOM. Those map onto real problems. Dynamic IDs invalidate selectors captured by pointing and clicking, shifting content breaks positional selectors, and elements inside a shadow root are not reachable by a query against the main document.

Reach for these when a specific selector works the moment you record it and fails on the next run. They are a debugging reference rather than a practice course, which is why they sit last.

What it does not cover: no realistic data and no record structure worth extracting.

Which site to use for a given question

If you want to	Start with
Learn selectors and crawl logic	books.toscrape.com
Check that a record count assertion works	books.toscrape.com, or /test-sites/pagination for 17 pages
Decide whether you need a rendering driver	quotes.toscrape.com/js
Tune a wait strategy	quotes.toscrape.com/js-delayed?delay=10000
Handle a load more button	/test-sites/load-more
Handle infinite scroll	/test-sites/scroll
Handle a CSRF-protected login	quotes.toscrape.com/login
Establish a session before extraction	/test-sites/website-state-setup-login
Work through a search form with pagination	scrapethissite.com/pages/forms
Test retry and backoff	httpbin.org/status/:code
Test timeouts and slow responses	httpbin.org/delay/:n
Work with realistic e-commerce markup	scrapeme.live/shop
Debug a selector that keeps breaking	uitestingplayground.com

A practice order that builds on itself

If you are learning rather than evaluating tools, the order you work through these sites in matters more than which one you start with. Each stage assumes the previous one produced a clean result.

Start on books.toscrape.com and extract all 1,000 records, not a sample. If you finish with 980, find the missing twenty before going further. Whatever swallowed them is the same class of problem that loses records in production, and it is far cheaper to find here.

Move to quotes.toscrape.com and run the raw HTML check before writing any extraction, on the default page and on /js. The habit being built is deciding whether you need a browser rather than assuming it.

Then the load more and scroll variants, with a count assertion already in place. This is the point where "the job completed" and "the data is complete" come apart, and where most scrapers quietly begin under-collecting.

Add session handling next. Use quotes.toscrape.com/login for CSRF tokens and cookie persistence, then scrapethissite.com/pages/advanced, which puts headers, cookies and session state in one exercise rather than three.

Finish on scrapeme.live/shop and httpbin. Markup nobody wrote for your benefit, followed by the retry and timeout behaviour that decides whether a long run survives a bad half hour.

Anyone comfortable with all five is still not ready for a hostile target, but they will be able to say which layer is failing when one pushes back.

Using these to evaluate a tool rather than learn a skill

If you are comparing scraping tools rather than learning to scrape, the four Web Scraper test site variants are more useful as a fixed benchmark than as a tutorial. The catalogue is identical across all four, so running the same extraction through each candidate tool shows you where its abstractions stop.

A practical sequence takes under an hour per tool:

1. Configure the extraction once against /test-sites/pagination and confirm you get every page. This establishes a baseline record count
2. Repoint the same extraction at /test-sites/load-more. Tools that treat pagination as link-following will silently return the first six items
3. Repeat against /test-sites/scroll. Some tools handle a button but not a scroll trigger
4. Run /test-sites/website-state-setup-login and see whether authentication is a first-class step or something you have to script around
5. Finish on scrapeme.live/shop to check the tool copes with markup it was not designed against

The output you want from that exercise is not "it worked" but a record count per step. A tool that returns 6 where it should return several hundred has not failed loudly, and that quiet under-collection is the behaviour you are testing for. The same logic applies once jobs are scheduled, which is why data quality rules and notifications matter more than run status.

What a quiet failure looks like in your own output

The recurring argument in this article is that scraping failures are usually silent. That is only useful if you know what silence looks like on your side. Each symptom below has a site above where you can reproduce it deliberately.

What you see	What it usually means	Where to reproduce it
Run completes, record count is a small round number	Pagination was never followed. You have page one	/test-sites/load-more
Run completes, records present, one field empty on every row	The selector targets a text node where the value sits in an attribute	books.toscrape.com star rating
Run completes, zero records, HTTP 200 throughout	Content is rendered client-side, or you are being served a login or consent page instead of the catalogue	quotes.toscrape.com/js and /test-sites/website-state-setup-login
Correct number of records, all of them empty	The container arrived before its contents. Your wait targets page load rather than data	quotes.toscrape.com/js-delayed
Record count correct at the start of a crawl, falling off later	Rate limiting or an expiring session partway through	httpbin.org/status/429 for the retry half
Run duration far below what the page count implies	Requests are failing fast and being counted as done	httpbin.org/status/500
Count correct, one field null on a subset of rows	A layout variant on some pages, such as discounted against standard products	scrapeme.live/shop

The pattern across all seven rows is that none of them raise an error. Every one produces a completed run and a file, which is why job status is a poor proxy for data quality.

What no sandbox will prepare you for

Every site above wants to be scraped. Production targets frequently do not, and the gap appears in ways that cannot be reproduced locally.

• Anti-bot systems that evaluate TLS fingerprints, header ordering and behavioural signals rather than request rate alone. No sandbox reproduces this, and no technique reliably defeats all of them. Why websites block scrapers covers what those systems are actually measuring
• Consent and geographic walls. A category URL returns 200 with a cookie banner and no products. From the transport layer the request succeeded; from the dataset's perspective it failed, and telling that apart from a bot challenge takes more than a status code
• IP reputation and rate limits, which become a proxy decision. Datacentre proxies are cheaper and faster, residential proxies are more likely to be accepted on sensitive targets, and neither guarantees access. The trade-off between the two depends on the target rather than on a general rule
• Layout drift. A class name changes, one field silently becomes null across a large share of records, and job status stays green because nothing threw
• Volume effects. Concurrency that is harmless against a sandbox can destabilise a smaller target, and logic that holds across 50 pages may not hold across 50,000

The way to close that gap is a staged run against the real target before anything is scheduled: a small sample first, with assertions on record count, null rate per field and value ranges, then a wider run once those hold.

What this list leaves out

Four categories were considered and deliberately excluded.

• Real public sites such as Wikipedia, Hacker News and IMDb, which are frequently recommended as practice targets. They are production services with terms of service and infrastructure costs, and practising against them builds the habit of treating someone else's server as a sandbox. That is the single worst habit to acquire early
• Pure UI automation demos such as saucedemo.com. They test clicking and form interaction rather than extraction, and nothing on them resembles a record structure
• TLS and certificate test endpoints. Useful when you are debugging a specific handshake or certificate failure, but that is a narrower problem than a practice site is for
• Sites that are no longer reliably up. Several sandboxes recommended in older lists now return errors intermittently. A practice site that fails at random teaches you nothing about your scraper, because you cannot tell your bug from its outage

FAQ

Which is the best site to practise web scraping on as a beginner?

books.toscrape.com, because the stated 1,000-record count gives you something to check your result against rather than assuming a clean run means a complete one. Move to the Web Scraper test sites once the parsing works, since that is where pagination stops being a link.

Are these sites legal to scrape?

They are published specifically for scraping practice, and books.toscrape.com states outright that it is a demo with randomly assigned prices and ratings. Treat that permission as specific to these sites rather than transferable. For any other target, robots.txt, terms of service and applicable law apply, and the considerations involved are set out in scraping public data, is it legal. None of that is a substitute for advice from a lawyer on your specific case.

Can I benchmark scraper performance on them?

Not usefully. They have small catalogues, low and stable latency, no anti-bot layer and no rate limiting, so throughput measured against them reflects your network more than your scraper. The same caution applies to published benchmark figures derived from sandboxes.

Do I need a headless browser for all of them?

No, and defaulting to one hides information you want. books.toscrape.com, the countries lesson on scrapethissite.com and /test-sites/pagination are fully server-rendered. Run the raw HTML check first: if the values are already in the initial response, an HTTP client is faster and cheaper.

Which practice site is closest to a real website?

scrapeme.live/shop, because it is an actual WooCommerce store rather than markup written for a tutorial. The class names, nesting and pagination behave the way a real theme does, including the inconvenient parts.

Can I test Web Scraper sitemaps against these?

Yes. The Web Scraper test sites exist for that and cover link pagination, a load more button, infinite scroll and a login-gated catalogue, which are the four sitemap behaviours most commonly configured incorrectly. The pagination selector documentation covers how each one is set up.

Next step

Once a sitemap returns the expected record count from the relevant test site, the remaining question is whether it behaves the same way outside your own browser. Build and verify the sitemap in the browser extension, then import it into Web Scraper Cloud and configure the driver, proxy and schedule there. Compare the first Cloud run against your local record counts before pointing it at a production target.