Scrapy Review: What You Use When the Meter Stops Making Sense

Every other scraper in this category has a pricing page. Credits, rows, records, executions, searches. Different words for the same idea, which is that someone counts what you take and charges you for it.

Scrapy has no pricing page, because there is nothing to price. It is BSD-3-Clause licensed, 64,000 stars on GitHub, and free in every sense the word carries.

You supply the machine and the code. Nobody counts anything.

When that actually matters

Most directory builders never need Scrapy, and that is worth saying up front rather than at the end.

If your niche has 3,000 businesses on Google Maps, use Apify and spend a few dollars. If you need a point-and-click tool, use Octoparse. If you need scheduled monitoring, use Browse AI. Those are better answers for the common case, and reaching for a Python framework instead is the classic mistake of choosing the powerful tool over the appropriate one.

Scrapy earns its place in three specific situations.

Volume that breaks the economics. Somewhere north of a few hundred thousand pages, per-row pricing stops being a rounding error and starts being a real line in the budget. Scrapy's cost at that volume is a server.

Sources nobody has built a scraper for. A regional trade association's member list. A government register. A specialist industry portal. There is no Apify Actor for these and there never will be, because the market is one person: you.

Structure that a visual tool cannot express. Scrape a listing page, follow every result into a detail page, follow a link from there into a third page, merge the three into one record, and only keep it if a field on the third page has a particular value. That is fifteen lines of Python and an afternoon of fighting with a point-and-click builder.

What Scrapy actually is

A complete scraping framework, not a library you assemble one yourself from.

You define a Spider, which is a class that says where to start, how to extract fields from a page, and which links to follow next. Scrapy handles everything around that: an asynchronous request engine that fetches many pages concurrently, automatic retries, request throttling, cookies and sessions, and a pipeline system for cleaning and storing what comes back.

That pipeline system is the part directory builders underrate. Cleaning is where scraped data actually gets usable, and Scrapy gives you a defined place to put that logic rather than a spreadsheet you edit by hand afterwards. Normalise phone numbers, title-case the names, drop rows with no website, deduplicate against a key, write to a database. All of it runs as the scrape runs.

Output goes to JSON, CSV or XML out of the box, or straight into a database through a custom pipeline.

The politeness settings, which you should not skip

Scrapy ships with configurable download delays, concurrency limits per domain, and an AutoThrottle extension that adjusts speed based on server response times.

Turn these on. A framework this fast against a small association's website is genuinely capable of causing problems for them, and the person running that site did nothing to deserve it. Set a delay of a couple of seconds. The scrape takes longer and nobody gets hurt.

This is also self-interested. Hammering a site is how you get blocked, and getting blocked is how a free tool starts costing you money in proxies.

Pricing

There is none. BSD-3-Clause, which is a permissive licence with essentially no obligations beyond keeping the copyright notice.

What you do pay for is adjacent, and it is worth being honest about:

Your time. If you already write Python, a first spider is an afternoon. If you do not, budget considerably more, or use an AI assistant to draft the spider and spend your time understanding rather than typing.

A server, if you want it scheduled. Running spiders on a schedule means somewhere for them to run. A small VPS covers it. Scrapyd, the project's own deployment daemon, handles running and scheduling spiders on that server.

Proxies, if you get blocked. Scrapy does not include a proxy pool. On defended targets you will need one, and that is where a service like ScraperAPI slots in underneath your spider. Note what that does to the argument: the free framework plus a paid unblocking layer is often the right combination, and it is still cheaper than per-row pricing at volume.

Where it does not fit

It is code. There is no interface, no visual builder, no recording your screen. If that sentence is a dealbreaker, the honest recommendation is Octoparse and there is no shame in it.

There is no scheduling built in. Scrapyd or cron handles it, but that is another thing to set up and maintain.

No proxy rotation or CAPTCHA solving. Both are solvable with middleware or an external service, and both are work you would not do on a managed platform.

And for Google Maps specifically, do not bother. Maps is heavily defended and the structured output is exactly what Apify and SerpApi already provide. Parsing Maps yourself is a project, not a task.

Common questions

Do I need to be a good Python programmer?

You need to be able to read Python and follow documentation. Scrapy's tutorial is genuinely good and gets you a working spider quickly. Modern AI assistants also write reasonable Scrapy code, which lowers the entry cost considerably, though you still need to understand enough to debug it when the selectors break.

Can it handle JavaScript-rendered pages?

Not on its own. Scrapy fetches HTML, so a page that builds its content in the browser returns nothing useful. The usual answer is scrapy-playwright, which adds browser rendering. It works, and it is slower and heavier, so use it only where you must.

How does it compare to Beautiful Soup?

Beautiful Soup parses HTML. Scrapy is the whole system: fetching, concurrency, retries, pipelines, export. If you have one page, use Beautiful Soup. If you have fifty thousand, use Scrapy.

Is it still maintained?

Yes, actively, and it has been going for well over a decade. 64,000 stars is a lot of people depending on it not breaking.

What licence is it?

BSD-3-Clause. Permissive, commercial use fine, no copyleft obligations.

Verdict

Scrapy is the answer to a question most directory builders do not have yet.

It is worth knowing about anyway, because the moment you look at a metered bill and realise the scrape you want costs more than a month of hosting, this is where you go. And it is worth knowing that the ceiling exists: no volume of data makes Scrapy more expensive, so there is no version of your directory that outgrows it.

If you write Python, read the tutorial and build one spider against a source you care about. If you do not, bookmark this and use a managed tool, and come back when a bill makes you reconsider.