README.md

WebCrawler

A super simple c# web crawler using System.Threading.Channels

Creates a publisher (Channels writer) and subscriber (Channels Reader) and uses a queue for managing urls. By default it restricts crawling to a single domain.