C# is still a popular backend programming language, and you might find yourself in need of it for scraping a web page (or multiple pages). In this article, we will cover scraping with C# using an HTTP request, parsing the results, and then extracting the information that you want to save. This method is common with basic scraping, but you will sometimes come across single-page web applications built in JavaScript such as Node.js, which require a different approach. We’ll also cover scraping these pages using PuppeteerSharp, Selenium WebDriver, and Headless Chrome.

• Best Web Scraper For Mac
• Scraping Tools

Note: This article assumes that the reader is familiar with C# syntax and HTTP request libraries. The PuppeteerSharp and Selenium WebDriver .NET libraries are available to make integration of Headless Chrome easier for developers. Also, this project is using .NET Core 3.1 framework and the HTML Agility Pack for parsing raw HTML.

The perfect web scraper doesn’t exist. However, modern technologies have allowed us to get pretty close. And with so many options out there, it’s easy to label any web scraper as “the best” or “most powerful”, but what are the must-have features that the best web scraper should have? Let’s break it all down. Ease of Use While ease of use might be a subjective matter, it is still.

Data Scraper is a simple and free web scraping tool for extracting data from a single page into CSV and XSL data files. It is a personal browser extension that helps you transform data into a clean table format. You will need to install the plugin in a Google Chrome browser. Best Free Web Scraper Software: The web scraper software that I like the most is Web Scraper. It is due to the fact that it is quite simple, lightweight, and easy to use. You can use it to scrape different types of data from the web like link, text, table, and many more such. Octoparse is the number one Automated Web Scraping Software. Octoparse is a cloud-based web scraper that helps the user easily extract any web data without coding. Octoparse is a new modern visual web data extraction software. Web scraping, sometimes called data scraping, data extraction, or web harvesting is merely the process of collecting data from websites and storing it on your local database or spreadsheets. Today, web scraping tools are necessary for the modern marketer.

Part I: Static Pages

Setup

If you’re using C# as a language, you probably already use Visual Studio. This article uses a simple .NET Core Web Application project using MVC (Model View Controller). After you create a new project, go to the NuGet Package Manager where you can add the necessary libraries used throughout this tutorial.

In NuGet, click the “Browse” tab and then type “HTML Agility Pack” to find the dependency.

Install the package, and then you’re ready to go. This package makes it easy to parse the downloaded HTML and find tags and information that you want to save.

Finally, before you get started with coding the scraper, you need the following libraries added to the codebase:

Making an HTTP Request to a Web Page in C#

Imagine that you have a scraping project where you need to scrape Wikipedia for information on famous programmers. Wikipedia has a page with a list of famous programmers with links to each profile page. You can scrape this list and add it to a CSV file (or Excel spreadsheet) to save for future review and use. This is just one simple example of what you can do with web scraping, but the general concept is to find a site that has the information you need, use C# to scrape the content, and store it for later use. In more complex projects, you can crawl pages using the links found on a top category page.

Using .NET HTTP Libraries to Retrieve HTML

.NET Core introduced asynchronous HTTP request libraries to the framework. These libraries are native to .NET, so no additional libraries are needed for basic requests. Before you make the request, you need to build the URL and store it in a variable. Because we already know the page that we want to scrape, a simple URL variable can be added to the HomeController’s Index() method. The HomeController Index() method is the default call when you first open an MVC web application.

Add the following code to the Index() method in the HomeController file:

Using .NET HTTP libraries, a static asynchronous task is returned from the request, so it’s easier to put the request functionality in its own static method. Add the following method to the HomeController file:

Let’s break down each line of code in the above CallUrl() method.

This statement creates an HttpClient variable, which is an object from the native .NET framework.

If you get HTTPS handshake errors, it’s likely because you are not using the right cryptographic library. The above statement forces the connection to use the TLS 1.3 library so that an HTTPS handshake can be established. Note that TLS 1.3 is deprecated but some web servers do not have the latest 2.0+ libraries installed. For this basic task, cryptographic strength is not important but it could be for some other scraping requests involving sensitive data.

This statement clears headers should you decide to add your own. For instance, you might scrape content using an API request that requires a Bearer authorization token. In such a scenario, you would then add a header to the request. For example:

The above would pass the authorization token to the web application server to verify that you have access to the data. Next, we have the last two lines:

These two statements retrieve the HTML content, await the response (remember this is asynchronous) and return it to the HomeController’s Index() method where it was called. The following code is what your Index() method should contain (for now):

The code to make the HTTP request is done. We still haven’t parsed it yet, but now is a good time to run the code to ensure that the Wikipedia HTML is returned instead of any errors. Make sure you set a breakpoint in the Index() method at the following line:

This will ensure that you can use the Visual Studio debugger UI to view the results.

You can test the above code by clicking the “Run” button in the Visual Studio menu:

Visual Studio will stop at the breakpoint, and now you can view the results.

If you click “HTML Visualizer” from the context menu, you can see a raw HTML view of the results, but you can see a quick preview by just hovering your mouse over the variable. You can see that HTML was returned, which means that an error did not occur.

Parsing the HTML

With the HTML retrieved, it’s time to parse it. HTML Agility Pack is a common tool, but you may have your own preference. Even LINQ can be used to query HTML, but for this example and for ease of use, the Agility Pack is preferred and what we will use.

Before you parse the HTML, you need to know a little bit about the structure of the page so that you know what to use as markers for your parsing to extract only what you want and not every link on the page. You can get this information using the Chrome Inspect function. In this example, the page has a table of contents links at the top that we don’t want to include in our list. You can also take note that every link is contained within an <li> element.

From the above inspection, we know that we want the content within the “li” element but not the ones with the tocsection class attribute. With the Agility Pack, we can eliminate them from the list.

We will parse the document in its own method in the HomeController, so create a new method named ParseHtml() and add the following code to it:

In the above code, a generic list of strings (the links) is created from the parsed HTML with a list of links to famous programmers on the selected Wikipedia page. We use LINQ to eliminate the table of content links, so now we just have the HTML content with links to programmer profiles on Wikipedia. We use .NET’s native functionality in the foreach loop to parse the first anchor tag that contains the link to the programmer profile. Because Wikipedia uses relative links in the href attribute, we manually create the absolute URL to add convenience when a reader goes into the list to click each link.

Exporting Scraped Data to a File

The code above opens the Wikipedia page and parses the HTML. We now have a generic list of links from the page. Now, we need to export the links to a CSV file. We’ll make another method named WriteToCsv() to write data from the generic list to a file. The following code is the full method that writes the extracted links to a file named “links.csv” and stores it on the local disk.

The above code is all it takes to write data to a file on local storage using native .NET framework libraries.

The full HomeController code for this scraping section is below.

Part II: Scraping Dynamic JavaScript Pages

In the previous section, data was easily available to our scraper because the HTML was constructed and returned to the scraper the same way a browser would receive data. Newer JavaScript technologies such as Vue.js render pages using dynamic JavaScript code. When a page uses this type of technology, a basic HTTP request won’t return HTML to parse. Instead, you need to parse data from the JavaScript rendered in the browser.

Dynamic JavaScript isn’t the only issue. Some sites detect if JavaScript is enabled or evaluate the UserAgent value sent by the browser. The UserAgent header is a value that tells the web server the type of browser being used to access pages (e.g. Chrome, FireFox, etc). If you use web scraper code, no UserAgent is sent and many web servers will return different content based on UserAgent values. Some web servers will use JavaScript to detect when a request is not from a human user.

You can overcome this issue using libraries that leverage Headless Chrome to render the page and then parse the results. We’re introducing two libraries freely available from NuGet that can be used in conjunction with Headless Chrome to parse results. PuppeteerSharp is the first solution we use that makes asynchronous calls to a web page. The other solution is Selenium WebDriver, which is a common tool used in automated testing of web applications.

Using PuppeteerSharp with Headless Chrome

For this example, we will add the asynchronous code directly into the HomeController’s Index() method. This requires a small change to the default Index() method shown in the code below.

In addition to the Index() method changes, you must also add the library reference to the top of your HomeController code. Before you can use Puppeteer, you first must install the library from NuGet and then add the following line in your using statements:

Now, it’s time to add your HTTP request and parsing code. In this example, we’ll extract all URLs (the <a> tag) from the page. Add the following code to the HomeController to pull the page source in Headless Chrome, making it available for us to extract links (note the change in the Index() method, which replaces the same method in the previous section example):

Similar to the previous example, the links found on the page were extracted and stored in a generic list named programmerLinks. Notice that the path to chrome.exe is added to the options variable. If you don’t specify the executable path, Puppeteer will be unable to initialize Headless Chrome.

Using Selenium with Headless Chrome

If you don’t want to use Puppeteer, you can use Selenium WebDriver. Selenium is a common tool used in automation testing on web applications, because in addition to rendering dynamic JavaScript code, it can also be used to emulate human actions such as clicks on a link or button. To use this solution, you need to go to NuGet and install Selenium.WebDriver and (to use Headless Chrome) Selenium.WebDriver.ChromeDriver. Note: Selenium also has drivers for other popular browsers such as FireFox.

Add the following library to the using statements:

Now, you can add the code that will open a page and extract all links from the results. The following code demonstrates how to extract links and add them to a generic list.

Notice that the Selenium solution is not asynchronous, so if you have a large pool of links and actions to take on a page, it will freeze your program until the scraping completes. This is the main difference between the previous solution using Puppeteer and Selenium.

Conclusion

Web scraping is a powerful tool for developers who need to obtain large amounts of data from a web application. With pre-packaged dependencies, you can turn a difficult process into only a few lines of code.

One issue we didn’t cover is getting blocked either from remote rate limits or blocks put on bot detection. Your code would be considered a bot by some applications that want to limit the number of bots accessing data. Our web scraping API can overcome this limitation so that developers can focus on parsing HTML and obtaining data rather than determining remote blocks.

Web scraping (also termed web data extraction, screen scraping, or web harvesting) is a technique of extracting data from the websites. It turns unstructured data into structured data that can be stored into your local computer or a database.

It can be difficult to build a web scraper for people who don’t know anything about coding. Luckily, there are tools available for people with or without programming skills. Also, if you're seeking a job for big data developers, using web scraper definitely raises your working effectiveness in data collection, improving your competitiveness. Here is our list of 30 most popular web scraping tools, ranging from open-source libraries to browser extension to desktop software.

Table of Content

1. Beautiful Soup

Who is this for: developers who are proficient at programming to build a web scraper/web crawler to crawl the websites.

Why you should use it: Beautiful Soup is an open-source Python library designed for web-scraping HTML and XML files. It is the top Python parsers that have been widely used. If you have programming skills, it works best when you combine this library with Python.

2. Octoparse

Who is this for: People without coding skills in many industries, including e-commerce, investment, cryptocurrency, marketing, real estate, etc. Enterprise with web scraping needs.

Why you should use it: Octoparse is free for life SaaS web data platform. You can use to scrape web data and turns unstructured or semi-structured data from websites into a structured data set. It also provides ready to use web scraping templates including Amazon, eBay, Twitter, BestBuy, and many others. Octoparse also provides web data service that helps customize scrapers based on your scraping needs.

3. Import.io

Who is this for: Enterprise looking for integration solution on web data.

Why you should use it: Import.io is a SaaS web data platform. It provides a web scraping solution that allows you to scrape data from websites and organize them into data sets. They can integrate the web data into analytic tools for sales and marketing to gain insight from.

4. Mozenda

Who is this for: Enterprise and business with scalable data needs.

Why you should use it: Mozenda provides a data extraction tool that makes it easy to capture content from the web. They also provide data visualization services. It eliminates the need to hire a data analyst.

5. Parsehub

Who is this for: Data analyst, Marketers, and researchers who lack programming skills.

Why you should use it: ParseHub is a visual web scraping tool to get data from the web. You can extract the data by clicking any fields on the website. It also has an IP rotation function that helps change your IP address when you encounter aggressive websites with anti-scraping techniques.

6. Crawlmonster

Who is this for: SEO and marketers

Why you should use it: CrawlMonster is a free web scraping tool. It enables you to scan websites and analyze your website content, source code, page status, etc.

7. ProWebScraper

Who is this for: Enterprise looking for integration solution on web data.

Best Web Scraper For Mac

Why you should use it: Connotate has been working together with Import.io, which provides a solution for automating web data scraping. It provides web data service that helps you to scrape, collect and handle the data.

8. Common Crawl

Who is this for: Researchers, students, and professors.

Why you should use it: Common Crawl is founded by the idea of open source in the digital age. It provides open datasets of crawled websites. It contains raw web page data, extracted metadata, and text extractions.

9. Crawly

Who is this for: People with basic data requirements.

Why you should use it: Crawly provides automatic web scraping service that scrapes a website and turns unstructured data into structured formats like JSON and CSV. They can extract limited elements within seconds, which include Title Text, HTML, Comments, DateEntity Tags, Author, Image URLs, Videos, Publisher and country.

10. Content Grabber

Who is this for: Python developers who are proficient at programming.

Why you should use it: Content Grabber is a web scraping tool targeted at enterprises. You can create your own web scraping agents with its integrated 3rd party tools. It is very flexible in dealing with complex websites and data extraction.

11. Diffbot

Who is this for: Developers and business.

Why you should use it: Diffbot is a web scraping tool that uses machine learning and algorithms and public APIs for extracting data from web pages. You can use Diffbot to do competitor analysis, price monitoring, analyze consumer behaviors and many more.

12. Dexi.io

Who is this for: People with programming and scraping skills.

Why you should use it: Dexi.io is a browser-based web crawler. It provides three types of robots — Extractor, Crawler, and Pipes. PIPES has a Master robot feature where 1 robot can control multiple tasks. It supports many 3rd party services (captcha solvers, cloud storage, etc) which you can easily integrate into your robots.

13. DataScraping.co

Who is this for: Data analysts, Marketers, and researchers who're lack of programming skills.

Why you should use it: Data Scraping Studio is a free web scraping tool to harvest data from web pages, HTML, XML, and pdf. The desktop client is currently available for Windows only.

14. Easy Web Extract

Who is this for: Businesses with limited data needs, marketers, and researchers who lack programming skills.

Scraping Tools

Why you should use it: Easy Web Extract is a visual web scraping tool for business purposes. It can extract the content (text, URL, image, files) from web pages and transform results into multiple formats.

15. FMiner

Who is this for: Data analyst, Marketers, and researchers who're lack of programming skills.

Why you should use it: FMiner is a web scraping software with a visual diagram designer, and it allows you to build a project with a macro recorder without coding. The advanced feature allows you to scrape from dynamic websites use Ajax and Javascript.

16. Scrapy

Who is this for: Python developers with programming and scraping skills

Why you should use it: Scrapy can be used to build a web scraper. What is great about this product is that it has an asynchronous networking library which allows you to move on to the next task before it finishes.

17. Helium Scraper

Who is this for: Data analysts, Marketers, and researchers who lack programming skills.

Why you should use it: Helium Scraper is a visual web data scraping tool that works pretty well especially on small elements on the website. It has a user-friendly point-and-click interface which makes it easier to use.

18. Scrape.it

Who is this for: People who need scalable data without coding.

Why you should use it: It allows scraped data to be stored on the local drive that you authorize. You can build a scraper using their Web Scraping Language (WSL), which is easy to learn and requires no coding. It is a good choice and worth a try if you are looking for a security-wise web scraping tool.

19. ScraperWiki

Who is this for: A Python and R data analysis environment. Ideal for economists, statisticians and data managers who are new to coding.

Why you should use it: ScraperWiki consists of 2 parts. One is QuickCode which is designed for economists, statisticians and data managers with knowledge of Python and R language. The second part is The Sensible Code Company which provides web data service to turn messy information into structured data.

20. Scrapinghub

Who is this for: Python/web scraping developers

Why you should use it: Scraping hub is a cloud-based web platform. It has four different types of tools — Scrapy Cloud, Portia, Crawlera, and Splash. It is great that Scrapinghub offers a collection of IP addresses covering more than 50 countries. This is a solution for IP banning problems.

21. Screen-Scraper

Who is this for: For businesses related to the auto, medical, financial and e-commerce industry.

Why you should use it: Screen Scraper is more convenient and basic compared to other web scraping tools like Octoparse. It has a steep learning curve for people without web scraping experience.

22. Salestools.io

Who is this for: Marketers and sales.

Why you should use it: Salestools.io is a web scraping tool that helps salespeople to gather data from professional network sites like LinkedIn, Angellist, Viadeo.

23. ScrapeHero

Who is this for: Investors, Hedge Funds, Market Analysts

Why you should use it: As an API provider, ScrapeHero enables you to turn websites into data. It provides customized web data services for businesses and enterprises.

24. UniPath

Who is this for: Bussiness in all sizes.

Why you should use it: UiPath is a robotic process automation software for free web scraping. It allows users to create, deploy and administer automation in business processes. It is a great option for business users since it helps you create rules for data management.

25. Web Content Extractor

Who is this for: Data analysts, Marketers, and researchers who're lack of programming skills.

Why you should use it:Web Content Extractor is an easy-to-use web scraping tool for individuals and enterprises. You can go to their website and try its 14-day free trial.

26. WebHarvy

Who is this for: Data analysts, Marketers, and researchers who lack programming skills.

Why you should use it: WebHarvy is a point-and-click web scraping tool. It’s designed for non-programmers. They provide helpful web scraping tutorials for beginners. However, the extractor doesn’t allow you to schedule your scraping projects.

27. Web Scraper.io

Who is this for: Data analysts, Marketers, and researchers who lack programming skills.

Why you should use it: Web Scraper is a chrome browser extension built for scraping data from websites. It’s a free web scraping tool for scraping dynamic web pages.

28. Web Sundew

Who is this for: Enterprises, marketers, and researchers.

Why you should use it: WebSundew is a visual scraping tool that works for structured web data scraping. The Enterprise edition allows you to run the scraping projects at a remote server and publish collected data through FTP.

29. Winautomation

Who is this for: Developers, business operation leaders, IT professionals

Why you should use it: Winautomation is a Windows web scraping tool that enables you to automate desktop and web-based tasks.

30. Web Robots

Who is this for: Data analysts, Marketers, and researchers who lack programming skills.

Why you should use it: Web Robots is a cloud-based web scraping platform for scraping dynamic Javascript-heavy websites. It has a web browser extension as well as desktop software, making it easy to scrape data from the websites.

Closing Thoughts

To extract data from websites with web scraping tools is a time-saving method, especially for those who don't have sufficient coding knowledge. There are many factors you should consider when choosing a proper tool to facilitate your web scraping, such as ease of use, API integration, cloud-based extraction, large-scale scraping, scheduling projects, etc. Web scraping software like Octoparse not only provides all the features I just mentioned but also provides data service for teams in all sizes - from start-ups to large enterprises. You can contact usfor more information on web scraping.