why using scrapy instead of beautiful soup or selenium
Answers
Answered by
0
Scrapy and BeautifulSoup have different purposes. BeautifulSoup is basically a library for parsing and extracting data from HTML.
Scrapy, on the other hand, is a framework that goes far beyond data extraction. Scrapy provides builtin solutions for the most common issues that you’ll find while scraping websites, such as: redirections, retrying certain types of requests, HTTP caching, filtering duplicated requests, auto-throttling to avoid overloading the servers, preserving sessions/cookies across multiple requests, among many other features.
Scrapy supports both CSS selectors and XPath expressions for data extraction. In fact, you could even use BeautifulSoup or PyQuery as the data extraction mechanism in your Scrapy spiders.
Long story short: if this project is not just a one-off scraper and if chances are that you’ll need to do web scraping in future projects, go with Scrapy. Even though your scraper might be starting simple, you’ll eventually hit some of the issues that Scrapy solves and learning Scrapy will quickly pay-off.
Scrapy, on the other hand, is a framework that goes far beyond data extraction. Scrapy provides builtin solutions for the most common issues that you’ll find while scraping websites, such as: redirections, retrying certain types of requests, HTTP caching, filtering duplicated requests, auto-throttling to avoid overloading the servers, preserving sessions/cookies across multiple requests, among many other features.
Scrapy supports both CSS selectors and XPath expressions for data extraction. In fact, you could even use BeautifulSoup or PyQuery as the data extraction mechanism in your Scrapy spiders.
Long story short: if this project is not just a one-off scraper and if chances are that you’ll need to do web scraping in future projects, go with Scrapy. Even though your scraper might be starting simple, you’ll eventually hit some of the issues that Scrapy solves and learning Scrapy will quickly pay-off.
Similar questions