URL Filters:

Following the Google Sitemaps MP3 Interview with John Mueller, I wanted to share with you a quick tip for using the GSiteCrawler software with DotNetNuke.

Make sure you listen to the full interview for tips on preparing your Google sitemaps. Following one of the tips, here's a filter list that I have created so far to remove links that Google does not require.

This filter list is currently used when crawling the DNN Creative website. It is particulalrly used to extract certain links such as the register / login / terms / privacy links on each page, plus extra links that are created and not relevant for Google if you are using Scott McCulloch's News Articles Module and ActiveModules Active Forums module.

Here is a list of the filters that I currently use in GSiteCrawler:
(in the ban URL tab)
(you will notice that some of these filters include the defaults that come with GSiteCrawler, I have left these on so that you can easily compare)

/_vti_bin/
/afsort/
/Categories/
/CategoryID/
/CategoryView/
/login
/NewsListing/
/PostComment
/privacy
/profile/
/register
/Search/
/sendemail/
/Syndication/
/terms
?
fsforum/
http://johannesmueller.com/_private

Hope this is useful

Date Metatags:

If you are using the Date Metatags with your DotNetNuke portal (as suggested in the interview).

When you make any content updates to a page and you have added the Date Metatag to the page, make sure you also update the Date Metatag, otherwise Google will not re-index your updated page....

Database Size:

Keep an eye on the size of the database that GSiteCrawler creates. - I was unaware for quite a while of just how big the database had grown. - It had reached a whole 1GB in size!

Quick solution:

Go to File and Compress Database

This reduced the 1GB file to just 2MB! - Quite a difference...

When GSiteCrawler compresses the database it also creates a .bak backup file of the original database, so if you need to free up the disk space of this large backup file (eg. in this case it was 1GB) - check that the database has been compressed without creating any errors (eg. that GSiteCrawler still operates correctly) and then re-click on:

File / Compress Database

This will then create a .bak file of the small 2MB file and delete the 1GB file.