What are the best tools for web scraping in 2022?

It's increasingly difficult these days to write scrapers that don't at some point need to execute JavaScript on a page - so you need to have a good browser automation tool on hand.

I'm really impressed by Playwright. It feels like it has learned all of the lessons from systems like Selenium that came before it - it's very well designed and easy to apply to problems.

I wrote my own CLI scraping tool on top of Playwright a few months ago, which has been a fun way to explore Playwright's capabilities: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

Beautiful Soup gets the job done. I made several app by using it.

[1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity)

[2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia)

[3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers)

[4] https://github.com/altilunium/wi-page (Scrap wikipedia to get most active contributors that contribute to a certain article)

[5] https://github.com/altilunium/arachnid (Web scraper, optimized for wordpress and blogger)

In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices.

[1] https://github.com/microsoft/playwright

[2] https://github.com/puppeteer/puppeteer

I built a tool called Browserflow (https://browserflow.app) that lets you automate any task in the browser, including scraping websites.

People love it for its ease-of-use because you can record actions via click-and-point rather than having to manually come up with CSS selectors. It intelligently handles lists, infinite scrolling, pagination, etc. and can run on both your desktop and in the cloud.

Grateful for how much love it received when it launched on HN 8 months ago: https://news.ycombinator.com/item?id=29254147

Try it out and let me know what you think!

Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes.

The only thing I wish was present was better support for RegExes. Bash and most unix tools don't support PCRE which can severely limiting. Plus, sometimes you want to process text as a whole vs line-by-line.

I would also recommend Python's sh[4] module if Shell scripting isn't your cup of tea. You get best of both worlds: faster dev work with Bash utils, and a saner syntax.

[1]: https://github.com/ericchiang/pup

[2]: https://csvkit.readthedocs.io/en/latest/

[3]: https://stedolan.github.io/jq/

[4]: https://pypi.org/project/sh/

curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that.

[1] https://github.com/lwthiker/curl-impersonate

We've built https://serpapi.com

We've invented the industry what you referring as "data type specific APIs"; APIs that abstract away all proxies issues, captcha solvings, various layouts support, even scrapping-related legal issues, and much more to a clean JSON response every single call. It was a lot of work but our success rate and response times are now rivaling non-scraping commercial APIs: https://serpapi.com/status

I think the next battle will be still legal despite all the wins in favor of scrapping public pages and common sense understanding this is the way to go. The EFF has been doing an amazing work in this world and we are proud to be a significant yearly contributor to the EFF.

As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control.

Scrapers are very simple, effective and probably one of the least fun things to build.

My http://heliumhq.com is open source and gives you a very simple Python API:

  from helium import *
  start_chrome('github.com/login')
  write('user', into='Username')
  write('password', into='Password')
  click('Sign in')

To get started:

  pip install helium

Also, you need to download the latest ChromeDriver and put it in your PATH.

Have fun :-)

Probably the best tool for scraping websites protected by services like cloudflare. https://github.com/ultrafunkamsterdam/undetected-chromedrive...

I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc.

The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.

estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.

It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.

estela has been recently published as OSS under the MIT license:

https://github.com/bitmakerla/estela

More details about it can be found in the release blog post and the official documentation:

https://bitmaker.la/blog/2022/06/24/estela-oss-release.html

https://estela.bitmaker.la/docs/

estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.

All kinds of feedback and contributions are welcome!

Disclaimer: I'm part of the development team behind estela :-)

Quick plug for running scrapers in GitHub Actions and writing the results back to a repository - which gives you a free way to track changes to a scraped resource over time. I call this "Git scraping" - I've written a whole bunch of notes about this technique here: https://simonwillison.net/series/git-scraping/

The polite package using R is intended to be a friendly way of scraping content from the owner. "The three pillars of a polite session are seeking permission, taking slowly and never asking twice."

https://github.com/dmi3kno/polite

Python is my work horse, if I need to scrape something from a site that is relaxed about scraping (most are). I have my own library of helper functions I've built up over the years. In simple cases I just regex out what I need, if I need a full DOM then I use JSDOM/node.

For sites that are "difficult" I remote control a real browser, GUI and all. I don't use Chrome headless because if there's e.g. a captcha I want to be able to fill it in manually.

For Ruby I recommend Medusa Crawler gem.

[1] https://github.com/brutuscat/medusa-crawler

Which I maintain as a fork of the unmaintained Anemone gem.

I’m not sure about “best” but I’ve been using Colly (written in Go) and it’s been pretty slick. Haven’t run in to anything it can’t do.

http://go-colly.org/

I don’t think the landscape has changed much since then. However, from my experience you should do everything possible to avoid a headless browser for scraping. It’s in the region of 10-100x slower and significantly more resource intensive, even if you carefully block unwanted requests (images, css, video, ads).

Obviously sometimes you have to go that route.

I have used the Apify SDK (now https://crawlee.dev/) in the past and found it very useful.

If the content you need is static, I like using node + cheerio [0] as the selector syntax is quite powerful. If there is some javascript execution involved however, I will fall back to puppeteer.

[0] - https://cheerio.js.org/

Can I piggyback on the question and ask what are people scraping these days?

I wrote my own webscraper: https://videlibri.de/xidel.html

The main purpose was to submit HTML forms. You just say in which input fields something should be written and then it does the other things (i.e. download the page, find all other fields and their default values, build a HTTP request from all of them and send that ).

The last 5 years, I spent updating the XPath implementation to XPath/XQuery 3.1. The W3C has put a lot new stuff in the new XPath versions like JSON support or higher order functions, for some reason they decided to turn XPath into a Turing-complete functional programming language.

I’ve got several years of experience of webscraping, mainly in python. Scrapy is the first choice for “basic websites” while playwright is used then things get difficult. I’m collecting my experience in using these tools in this “web scraping open knowledge project” on github (https://github.com/reanalytics-databoutique/webscraping-open...) and on my substack (http://thewebscraping.club/) for longer free content

I have had some luck running puppeteer in a nodejs app hosted at glitch.com. I spring for the (cheap) paid hosting and get several containers for dev/test/prod, web based ide. Obviously, this would only scale to a point. In my case I just need a single client automating interaction with a single site. If I really needed scale, I'd probably use one of the services listed elsewhere.

Of course, if you don't need a full javascript-enabled browser parse, consider alternatives first: simple HTTP requests, API, RSS, etc.

For simple scraping where the content is fairly static, or when performance is critical, I will use linkedom to process pages.

https://github.com/WebReflection/linkedom

When the content is complex or involves clicking, Playwright is probably the best tool for the job.

https://github.com/microsoft/playwright

Has anybody created anything similar to Portia for scraping? I'd love to self host or pay a nominal fee to allow my team to create / adjust scrapers via a UI

Obviously scraping logic using puppeteer, but there are many other tooling aspects that are critical to bypass bot prevention.

one is signature / fingerprinting emulation. It helps to run the bot in a real browser and export the fingerprint (e.g. UA, canvass, geoloc etc) into JS object . Add noise to the data too.

Simulate residential IPs by routing through a residential proxy. If you run bots from cloud you will get blocked.

I’ve built a lot of tools utilizing web scraping most recently https://GitHub.com/Jawerty/myAlgorithm and https://metaheads.xyz I think the more control you have over the tools the better if you know your way around css selectors and selenium you can do anything web scraping. Selenium can seem hefty but there are plenty of ways to optimize for resource intensity; look up selenium grid. Overall, don’t be afraid of browser automation you can Always find a way to optimize. The real difficulty is freshness of html. This you can fix by being smart about time stamps and caching. If you have the same data you’re scraping consistently…don’t do that. Also if there’s a frontend in your application dependent on scraped data NEVER use your scraping routines as a direct feed, store data whenever you scrape.

i hate to be that guy, but “it depends”

scrapy is still king for me (scrapy.org). there are even packages to use headless browsers for those awful javascript heavy sites

however, APIs and RSS are still in play, and that does not require a heavy scraper. I am building vertical industry portals, and many of my data rollups consume APIs and structured XML/RSS feeds from social and other sites.

Many years ago I wrote a scraper-module for a scripting language that exposed a fake DOM to an embedded JS-engine, spidermonkey. The DOM was just an empty object graph, readable both from the scripting language and inside the JS context. The documents were parsed by libxml2 and the resulting DOMs were not identical to mozilla's, for example. But fast and efficient.

The purpose was to enable "live interactive" scraping of forms/js/ajax sites, with a web frontend controlling maybe 10 scrapers for each user. When that project fell through, I stopped maintaining it and the spidermonkey api has long since moved on.

It works for simple sites that don't require the DOM to actually do anything (for example triggering images to load with some magic url). But many simple DOM behaviours can be implemented.

It depends on what you are trying to accomplish, but I think a combination of Puppeteer and JSDOM or Cheerio should take you far. Where it gets complex is when you need to do things such as rotating IPs, but in my experience, that's only needed if you're engaging in a heavy scraping workload.

Puppeteer + JSDOM is what I used to build https://www.getscrape.com, which is a high-level web scraping API. Basically, you tell the API if you want links, images, texts, headings, numbers, etc; and the API gets all that stuff for you without the need to pass selectors or parsing instructions.

In case anyone here wants something straightforward. It works well to build generic scraping operations.

I'm working on a personal project that involves A LOT of scraping, and through several iterations I've gotten some stuff that works quite well. Here's a quick summary of what I've explored (both paid and free):

* Apify (https://apify.com/) is a great, comprehensive system if you need to get fairly low-level. Everything is hosted there, they've got their own proxy service (or you can roll your own), and their open source framework (https://github.com/apify/crawlee) is excellent.

* I've also experimented with running both their SDK (crawlee) and Playwright directly on Google Cloud Run, and that also works well and is an order-of-magnitude less expensive than running directly on their platform.

* Bright Data nee Luminati is excellent for cheap data center proxies ($0.65/GB pay as you go), but prices get several orders of magnitude more expensive if you need anything more thorough than data center proxies.

* For some direct API crawls that I do, all of the scraping stuff is unnecessary and I just ping the APIs directly.

* If the site you're scraping is using any sort of anti-bot protection, I've found that ScrapingBee (https://www.scrapingbee.com/) is by far the easiest solution. I spent many many hours fighting anti-bot protection doing it myself with some combination of Bright Data, Apify and Playwright, and in the end I kinda stopped battling and just decided to let ScrapingBee deal with it for me. I may be lucky in that the sites I'm scraping don't really use JS heavily, so the plain vanilla, no-JS ScrapingBee service works almost all of the time for those. Otherwise it can get quite expensive if you need JS rendering, premium proxies, etc. But a big thumbs up to them for making it really easy.

Always looking for new techniques and tools, so I'll monitor this thread closely.

We’ve built a freemium cloud RPA software focused on web scraping and monitoring, called Browse AI.

https://www.browse.ai

It lets you train a bot in 2 minutes. The bot will then open the site with rotating geolocated ip addresses, solve captchas, click on buttons and scroll and fill out forms, to get you the data you need.

It’s integrated with Google Sheets, Airtable, Zapier, and more.

We have a Google Sheets addon too which lets you run robots and get their results all in a spreadsheet.

We have close to 10,000 users with 1,000+ signing up every week these days. That made us raise a bit of funding from Zapier and others to be able to scale quicker and build the next version.

For a particular type of scraping, we wrote SSScraper on top of Colly and it works really well:

https://github.com/gotripod/ssscraper/

* Shameless plug *: our super-easy feed builder at New Sloth (formerly Feedity) - https://newsloth.com combines a scraper and data transformer, which helps create custom RSS feeds for any public webpage. Our API can auto-magically detect relevant articles in most cases. The platform includes an integrated feed reader and clusterer/deduplicator, specially aimed for knowledge workers with hundreds and thousands of feeds to monitor daily.

Not a full fledged scraper but IDS[1] has great heuristics to figure relevant content/information behind HTML code therefore lesser/no iterations needed in case frontend code changes.

Would be cool to reverse engineer it and probably plug it into some JS rendering testing solution (say Puppeteer, etc.)

[1] https://chrome.google.com/webstore/detail/instant-data-scrap...

Normalize Rest calls so programmers don’t have to rely on webscrapers, selenium and other flaky methods of retrieving data.

Web scraping is fun, but in production it’s an absolute joke.

I am probably in the minority, but I try to outsource scraping whenever possible. It's too much grunt work: you have to constantly baby sit the crawlers that break because websites keep changing.

Personally, I use Indexed (https://www.indexedinc.com) because they are technical and reliable, although there are many other providers out there..

Is there any form of markup / library that would allow me to access a file-tree similar to what show's up in the chrome "inspect" "sources" tab? I'm working on a system to extract m3u8 files from websites. Haven't found a good way to do this yet a few years since my last project that required scraping with a headless browser.

Have to appreciate the irony of someone's SEO spam submission (submitter works for a company selling scraping services) being SEO spammed in the comments...

>Thanks for the links. And I read too. I see a lot of useful stuff that I will use for my site https://los-angeles-plumbers.com/

Selenium via Python is really useful too if you need to do a bit more (e.g clicks) than just fetching the html from the page.

Scrapy's crawling and CSS/xpath selectors are fine. But I'm annoyed about the pipeline after that. Especially to get the data into a SQLite database. I wish cleaning up the data was a series of transformations on SQL tables instead of a bunch of work on Python models.

A good no-code solution is https://simplescraper.io. Leans towards non-developers but there's an API too.

I’m biased since I’m an owner of a web scraping agency (https://webscrapingsolutions.co.uk/). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy (https://github.com/scrapy/scrapy), Redis, PostgreSQL. for the following reasons:

[1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of training. There are a lot of guides for beginners.

[2] Lots of features are already implemented and open-source, you won’t have to waste time & money on them.

[3] There is a strong community that can help with most of the questions (I don't think any other alternative has that).

[4] Scrapy developers are cheap. You will only need junior+ to middle level software engineers to pull out most of the projects. It’s not rocket since.

[5] Recruiting is easier: - there are hundreds of freelancers with relevant expertise - if you search on LinkedIn - there are hundreds of software developers that have worked with Scrapy in the past, and you don’t need that many - you can grow expertise in your own team quickly - developers are easily replaceable, even on larger projects - you can use the same developers on backend tasks.

[6] You don’t need a DevOps expertise in your web scraping team because Scrapy Cloud (https://www.zyte.com/scrapy-cloud/) is good and cheap enough for 99% of the projects.

[7] If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd.

[8] The entire ecosystem is well-well-maintained and steadily growing. You can integrate a lot of 3-rd party services into your project within hours: proxies, captcha solving, headless browsers, HTML parsing APIs.

[9] It’s easy to integrate your own AI/ML models into the scraping workflow.

[10]. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis.

[11] Commercial support is available. There are several companies that can develop you an entire project or take over an existing one - if you don’t have the time/don’t want to do it on your own.

We have built dozens of projects in multiple industries:

- news monitoring

- job aggregators

- real estate aggregators

- ecommerce (anything from 1 website, to monitoring prices on 100k+ domains)

- lead generation

- search engines in a specific niche (SEO, pdf files, ecommerce, chemical retail)

- macroeconomic research & indicators

- social media, NFT marketplaces, etc

So, most of the projects can be finished using these tools.

I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.

Python, requests, BeautifulSoup, lxml, BrightData proxy provider. If necessary, async (if you’re scraping multiple pages) and Pyppeteer (if scraping JS-heavy pages).

Try https://webscraping.ai/ if you need rotating proxies and JS rendering

https://www.page2api.com/ - powerful API, easy to use, pay-as-you-go.

I use Scraping Fish API: https://scrapingfish.com/

You can also use the common crawl dataset.

https://commoncrawl.org/

If no javascript automation - python's asyncio, regex and aiohttp can scale really fast for simple tasks.

it depends. for no-code solution, please check [powerpage-web-crawler](https://github.com/casualwriter/powerpage-web-crawler) for crawling blog/posts.

Playwright is imho great if you want to avoid the common pitfalls of most scrapers.

if you are tech enough to find the query-selector of the elements, here is a great tool.

Great things is, it have support for Zapier, webhooks and API access too.!

https://browserbird.com

Perl + Mojo::DOM, or Perl + Playwright or Selenium when JavaScript is required

Selenium with custom geckodriver/chromedriver (to counter antibot hacks)

if you are looking for pre-made tools and dont want to write any code, check our https://webautomation.io

Scrapy for low level stuff plus some tool that can run Javascript

I use ScrapingBee (with Python, requests, & BeautifulSoup).

In the Node realm; you can't beat undici + cheerio.

It’s still curl and selenium.

Selenium is very good

python requests python bs4 buy some proxy and go!

e-scraper.com

powershell