Iron Webscraper facilitates C# development of screen scraping and data-mining apps with its VB/C# interface enabling developers to write workflows that emulate human browsing behavior. This .Net DLL and Nuget package simplifies web scraping.
As a web scraping tool for C#, Iron WebScraper is a fantastic option for developers using the .Net 4.5 and Core platform. With this library, developers can efficiently and cleanly extract content from a variety of web applications, parsing it into JSON, spreadsheets, C# objects, or even SQL.
Using simple C# and Linq code, this library can extract pages using set-by-step workflows, scraping and parsing html, javascript, xml, RSS, pdfs, and office documents from both the internet and local intranets easily. The result is well-structured data and a more understandable and debuggable web-scraping application.
The C# Web Scraping Library is designed to be exceptionally polite, ensuring that domain or IP addresses do not have too many concurrent requests. It intelligently throttles both the client and server side by monitoring excessive CPU usage and gradually slowing down to an appropriate pace. Also, it can adhere to robots.txt directives, including bot-specific crawl rates and limitations.
The exact URLs and content types to be extracted can be set using logical workflows and Regex/wildcard rules, which is intended to be used as an aid to screen-scraping. Iron WebScraper also employs identity control, automatically managing threads, rate limits, urls, duplicates, retries, proxies, headers, and cookies into a virtual browser army that can mimic human behavior, client buttons, fill in forms, or even log in behind security walls. This feature is beneficial for migrating legacy systems, populating enterprise search facilities, or for competitive analysis purposes.
The full documentation and support for the C# WebScraper are readily available, with downloadable DLLS at /csharp/webscraper/ in addition to links to a .Net 4.5+ Nuget package with full Azure and Mono compatibility. This web-scraping tool is undoubtedly an excellent addition to any developer's toolset.
Using simple C# and Linq code, this library can extract pages using set-by-step workflows, scraping and parsing html, javascript, xml, RSS, pdfs, and office documents from both the internet and local intranets easily. The result is well-structured data and a more understandable and debuggable web-scraping application.
The C# Web Scraping Library is designed to be exceptionally polite, ensuring that domain or IP addresses do not have too many concurrent requests. It intelligently throttles both the client and server side by monitoring excessive CPU usage and gradually slowing down to an appropriate pace. Also, it can adhere to robots.txt directives, including bot-specific crawl rates and limitations.
The exact URLs and content types to be extracted can be set using logical workflows and Regex/wildcard rules, which is intended to be used as an aid to screen-scraping. Iron WebScraper also employs identity control, automatically managing threads, rate limits, urls, duplicates, retries, proxies, headers, and cookies into a virtual browser army that can mimic human behavior, client buttons, fill in forms, or even log in behind security walls. This feature is beneficial for migrating legacy systems, populating enterprise search facilities, or for competitive analysis purposes.
The full documentation and support for the C# WebScraper are readily available, with downloadable DLLS at /csharp/webscraper/ in addition to links to a .Net 4.5+ Nuget package with full Azure and Mono compatibility. This web-scraping tool is undoubtedly an excellent addition to any developer's toolset.
What's New
Version 4.0.4.2:
'IronWebScraper' is a new Web Scraping Library for the C# / .Net programming platform.
Release features include:
- Read structured content from websites for population of databases, search indexes & applications.
- Submission to Nuget.org : https://www.nuget.org/packages/IronWebScraper/