sjdirect 

Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to... More information

A robots.txt parser written in c#. This is an unofficial fork of NRobots.txt on codeplex with extended functionality. For full documentation see https://nrobots.codeplex.com/.

A powerful C# web crawler that makes advanced crawling features easy to use. AbotX builds upon the open source Abot C# Web Crawler by providing a powerful set of wrappers and extensions.

Search engines and social networks are always trying to crawl your pages, but they only see the javascript tags... We render your javascript in a browser, save the static HTML, and you return that to the... More information

C# common libs, utilities and helpers.