Web Crawler - Source Code

Back to Web Crawler main page

Web Crawler component

The main procedure is 'Crawl'. We call it recursively for each page we crawl.
First, we browse the desired url and get the HTML out.
We also get the original buffer of the page and then content type:

GetHTMLFromSite(url, ref html, out encoding, string.Empty, out buffer, ref contentType);

Then we need to convert the HTML to XML so we can get a readable structure. We create the HTML Parser object:

Noviway.HTMLParser.Parser parser = new Noviway.HTMLParser.Parser(logger, html);

parser.Process();

After we processed the data we can convert it to XML:

Noviway.HTMLParser.HTMLDocument doc = new Noviway.HTMLParser.HTMLDocument(logger, parser.Tags, encoding);

doc.Process();

doc.CreateXml(out xmlDoc);

Then we raise the NewPage event. We use a structure called WebPage to store the page data.
This structure is passed to the callback function ( the event ).

Now, we've got the XML document to parse.
We go over all anchor tags:

ArrayList list = doc.GetElementsByTagName("a");

Demo

Create the crawler object. Set the maximum crawling levels, WebPage parameters and the callback function:

Noviway.WebCrawler.Crawler crawler = new Noviway.WebCrawler.Crawler();

crawler.MaxCrawlingLevels = 3
crawler.SaveBuffer = true;
crawler.SaveHTML = true;
crawler.NewPageEvent = new Noviway.WebCrawler.Crawler.NewPageCallback(NewPageEvent);
crawler.Crawl("http://www.noviway.com");

Create a callback function. This function will receive notifications about new web pages crawled.

private void NewPageEvent(Noviway.WebCrawler.Crawler.WebPage page, int level)

That's it, you can do whatever you please with the WebPage object you get.

NAVIGATION MENU

Statistics
TCP/IP Sniffer ( Basic )
Thread Pooling
HTTP Browsing
Winsock - Client & Server
Web Crawler
Binary Search
HTML To XML Converter