Hello,

I am looking for help in how to implement a robots.txt file.  In case it matters, my site is running DNN5 and is configured to use user-friendly URLs.  I have read conflicting information about what should be in the robots.txt file.  Some say it should be very simple, while others say it should be complex.  My site was recently moved from FrontPage to DNN.  Since the move 10 days ago, Google has not been able to index any of my aspx pages.  By this I mean www.MyDomain.com has a Page Ranking while www.MyDomain.com/home.aspx and all other pages do not.  Here is what I was planning on using for the robots file:

********************************************

User-agent: MS Search 4.0 Robot
Disallow: /

User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft
Disallow: /

User-agent: *
Disallow: /*/ctl/
Disallow: /admin/
Disallow: /App_Browsers/
Disallow: /App_Code/
Disallow: /App_Data/
Disallow: /App_GlobalResources/
Disallow: /bin/
Disallow: /Components/
Disallow: /Config/
Disallow: /controls/
Disallow: /DesktopModules/
Disallow: /Documentation/
Disallow: /images/
Disallow: /Install/
Disallow: /js/
Disallow: /Portals/
Disallow: /Providers/
Disallow: /Resources/

The above 18 items on the Disallow list would be repeated again for the following 4 agents:
User-agent: msnbot
User-agent: Slurp
User-agent: Googlebot
User-agent: Yahoo Pipes 1.0

********************************************

Here are my questions:

1. Is this what I should use?  If not, please suggest edits.

2. Will the above allow Google, Yahoo, etc. to index all of my dynamic aspx pages?  Or do I need to open up /Portals/ or some other directory folder? 

3. I use several third-part modules, most notably NewsArticles for a blog and to list News Items.  Will this content be visible to robots with the above or do I need to open up other folders?

4. Finally, please confirm that I should place this file in the root directory of HTTPdocs folder.

Thanks so much for helping me to wrap this up!

Chip

Here is one helpful reference.

Tom Kraak is a reigning expert on SEO and DotNetNuke.  You'll find more useful information at seablick.com.