As you are searching for the best open source web crawlers, you surely know they are a great source of data for analysis and data mining.
Internet crawling tools are also called web spiders, web data extraction software, and website scraping tools.
The majority of them are written inJava, but there is a good list of free and open code data extracting solutions inC#, C, Python, PHP, and Ruby. You can download them on Windows, Linux, Mac or Android.
Web content scraping applications can benefit your business in many ways.They collect content from different public websites and deliver the data in a manageable format.They help you monitoring news, social media, images, articles, your competitors, and etc.
On this page:
- 10 of the best open sourceweb crawlers.
- How to choose open sourceweb scraping software? (with an Infographic in PDF)
1. Scrapy
Scrapy is an open source and collaborative framework for data extracting from websites.It is a fast, simple but extensible tool written in Python. Scrapy runs on Linux, Windows, Mac, and BSD.
Itextracting structured data that you can use for many purposes and applications such as data mining, information processing or historical archival.
Scrapy was originally designed for web scraping. However, it is also used to extract data using APIs or as a web crawler for general purposes.
Key features and benefits:
- Built-in support for extracting data from HTML/XML sources using extended CSS selectors and XPath expressions.
- Generating feed exports in multiple formats (JSON, CSV, XML).
- Built on Twisted
- Robust encoding support and auto-detection.
- Fast and simple.
2.Heritrix
Heritrix is one of the most popular free and open-source web crawlers in Java. Actually, it is an extensible, web-scale, archival-quality web scraping project.
Heritrix is a very scalable and fast solution. You can crawl/archive a set of websites in no time. In addition, it is designed to respect the robots.txt exclusion directives and META robots tags.
Runs on Linux/Unixlike and Windows.
Key features and benefits:
- HTTP authentication
- NTLM Authentication
- XSL Transformation for link extraction
- Search engine independence
- Mature and stable platform
- Highly configurable
- Runs from any machine
WebSphinix is a great easy to use personal and customizable web crawler. It is designed for advanced web users and Java programmers allowing them to crawl over a small part of the web automatically.
This web data extraction solution also is a comprehensive Java class library and interactive development software environment.WebSphinix includes two parts: the Crawler Workbench and the WebSPHINX class library.
The Crawler Workbench is a good graphical user interface that allows you to configure and control a customizable web crawler.The library provides support for writing web crawlers in Java.
WebSphinix runs onWindows, Linux, Mac, and Android IOS.
Key features and benefits:
- Visualize a collection of web pages as a graph
- Concatenate pages together for viewing or printing them as a single document
- Extract all text matching a certain pattern.
- Tolerant HTML parsing
- Support for the robot exclusion standard
- Common HTML transformations
- Multithreaded Web page retrieval
4. Apache Nutch
When it comes to best open sourceweb crawlers,Apache Nutch definitely has a top place in the list.Apache Nutch is popular as a highly extensible and scalable open source code web data extraction software project great for data mining.
Nutch can run on a single machine but a lot of its strength is coming from running in a Hadoop cluster.
Many data analysts and scientists, application developers, and web text mining engineers all over the world use Apache Nutch.
Apache Nutch is a cross-platform solution written in Java.
Key features and benefits:
- Fetching and parsing are done separately by default
- Supports a wide variety of document formats: Plain Text, HTML/XHTML+XML, XML, PDF, ZIP and many others
- Uses XPath and namespaces to do the mapping
- Distributed filesystem (via Hadoop)
- Link-graph database
- NTLM authentication
5. Norconex
A great tool for those who are searching open source web crawlers forenterprise needs.
Norconex allows you to crawl any web content. You can run this full-featured collector on its own, or embed it in your own application.
Works on any operating system. Can crawl millions on a single server of average capacity. In addition, it has many content and metadata manipulation options.Also, it can extract page “featured” image.
Key features and benefits:
- Multi-threaded
- Supports different hit interval according to different schedules
- Extract text out of many file formats (HTML, PDF, Word, etc.)
- Extract metadata associated with documents
- Supports pages rendered with JavaScript
- Language detection
- Translation support
- Configurable crawling speed
- Detects modified and deleted documents
- Supports external commands to parse or manipulate documents
- Many others
6.BUbiNG
BUbiNG will surprise you.It is a next-generation open source web crawler. BUbiNG is a Java fully distributed crawler (no central coordination). It is able to crawl several thousands pages per second. Collect really big datasets.
BUbiNG distribution is based on modern high-speed protocols so to achieve very high throughput.
BUbiNG provides massive crawling for the masses. It is completely configurable, extensible with little efforts and integrated with spam detection.
Key features and benefits:
- High parallelism
- Fully distributed
- Uses JAI4J, a thin layer over JGroups that handles job assignment.
- Detects (presently) near-duplicates using a fingerprint of a stripped page
- Fast
- Massive crawling.
7. GNU Wget
GNU Wget is a free and open source software tool written in C for retrieving files using HTTP, HTTPS, FTP,and FTPS.
The most distinguishing feature is that GNU Wget has NLS-based message files for many different languages.In addition, it can optionally convert absolute links in downloaded documents to relative documents.
Runs on most UNIX-like operating systems as well as Microsoft Windows. GNU Wget is a powerful website scraping tool with a variety of features.
Key features and benefits:
- Can resume aborted downloads, using REST and RANGE
- Can use filename wild cards and recursively mirror directories
- Supports HTTP proxies
- Supports HTTP cookies
- Supports persistent HTTP connections
- Unattended / background operation
Arachnode.net is for those who are looking for open source web crawlers in is a C#. Arachnode.net is a class library which downloads content from the internet, indexes this content and provides methods to customize the process.
You can use the tool for personal content aggregation or you can use the tool for extracting, collecting and parse downloaded content into multiple forms.Discovered content is indexed and stored in Lucene.NET indexes.
Arachnode.net is a good software solution for text mining purposes as well as for learning advanced crawling techniques.
Key features and benefits:
- .NET architecture – the most comprehensive open source C#.
- Configurable rules and actions
- Lucene.NET Integration
- SQL Server and full-text indexing
- .DOC/.PDF/.PPT/.XLS Indexing
- HTML to XML and XHTML
- Full JavaScript/AJAX Functionality
- Multi-threading and throttling
- Respectful crawling
- Analysis services
OpenSearchServer is an open source enterprise class search engine and web crawling software.It is a fully integrated and very powerful solution. One of the best solutions out there.
OpenSearchServer has one of the high rated reviews on the internet.It is packed with a full set of search functions and allows you to build your own indexing strategy.
The web crawler includes inclusion or exclusion filters with wildcards, HTTP authentication, screenshot, sitemap, Etc.It is written in C, C++, and Java PHP and is a cross-platform solution.
Key features and benefits:
- A fully integrated solution
- The crawlers can index everything
- Full-text, boolean and phonetic search
- 17 language options
- Automatic classifications
- Scheduling for periodic tasks
- Parsing: Office documents ( such as Word, Excel, Powerpoint), OpenOffice documents, PDF files, Web pages (HTML), RTF, plain text, audio files, metadata images and etc.
10.Nokogiri
If you use Ruby,Nokogiri could be your solution.Nokogiri can transform a webpage into a ruby object. In addition, it makes all the web crawling process really easy and simple.
Nokogiri is an HTML, XML, SAX, and Reader parser. It has many features and the ability to search documents via XPath or CSS3 selectors is one of the best.
Nokogiri is a large library and provides example usages for parsing and examining a document.This data extraction software runs on Windows, Linux, Mac OS, Ubuntu.
Key features and benefits:
- XML/HTML DOM parser which handles broken HTML
- XML/HTML SAX parser
- XML/HTML Push parser
- XPath 1.0 support for document searching
- CSS3 selector support for document searching
- XML/HTML builder
- XSLT transformer
How to choose the best open source website crawler?
Crawling or scraping data software tools are becoming more and more popular. Hundreds of options have become available with different functionality and scalability.
Choosing the right option can be a tricky business. Here are sometips to help you find out the right open source web scraping software for your needs.
- Scalability
The web data extraction solution that you choose should be scalable. If your data needs aregrowing, the crawling tool shouldn’t slow you down. Your future data requirements should be covered.
This means the website crawler architecture should permit adding extra machines and bandwidth to handle future scaling up.
- Distributed web crawling
It means all downloaded pages have to be distributed among many computers (even hundreds of computers) in fraction of seconds.
In other words, the web data extraction software should have the capability to perform in a distributed way across multiple machines.
- Robustness
Robustness refers to the web scraper ability to notget trapped in a large number of pages.
Website scrapers must be stable and not fall in the trap generated by many web servers which trick the crawlers to stop working while fetching an enormous number of pages in a domain.
- Politeness
Politeness is a must for all of the open source web crawlers.Politeness means spiders and crawlers must not harm the website.To be polite a web crawler should follow the rules identified in the website’s robots.txt file.
Also, your web crawler should have Crawl-Delay and User-Agent header.Crawl-Delay refers to stopping the bot from scraping website very frequently. When a website has too many requests that the server cannot handle, they become unresponsive and overloaded.
User-Agent header allows you to include your contact details (such as email and website) in it. Thus the website owner will contact you in case you are ignoring the core rules.
- Extensible
Open source web crawlers should be extensible in many terms. They have to handle new fetch protocols, new data formats, and etc. In other words, the crawler architecture should be modular.
- Data delivery formats
Ask yourself what data delivery formats you need.Do you need JSON format? Then choose a web data extraction software that delivers the data in JSON. Of course, the best choice is to find one that delivers data in multiple formats.
- Data quality
As you might know,the scraped data is initially unstructured data (see unstructured data examples).You need to choose a software capable of cleaning the unstructured data and presenting it in a readable and manageable manner.
It doesn’t need to be a data cleansing software but should take care of cleaning up and classifying the initial data into useful data for you.
Conclusion
Scraping or extracting information from a website is an approach applied by a number of businesses that need to collect a large volume of data related to a particular subject.
All of the open source web crawlers have their own advantage as well as cons.
You need to carefully evaluate the web scrapers and then choose one according to your needs and requirement.
For example, Scrapy is faster and very easy to use but it is not as scalable as Heritrix, BUbiNG, and Nutch. Scrapy is also an excellent choice for those who aim focused crawls.
Heritrix is scalable and performs well in a distributed environment. However, it is not dynamically scalable. On the other hand, Nutch is very scalable and also dynamically scalable through Hadoop.Nokogiri can be a good solution for those that want open source web crawlers in Ruby. And etc.
If you need more open source solution related to data, then our posts about best open source data visualization software and best open source data modeling tools, might be useful for you.
Which are your favorite open source web crawlers? What data do you wish to extract?
Download the following infographic in PDF:
