SiteMap and Robots txt

[Maxx]

OK the site Map, this is something that seems to be the talk of town now when it comes to getting your site indexed on G00gle and other popular search engines, this allows the spiders to easily read and index the pages of your community site. there are now many free sitemap generators tools out there but I found this one that will generate up to 500page for you free, then you can add this xml file to your root directory! to read more on the subject you may follow this link cut and paste  http://www.xml-sitemaps.com

At this site you will be given information on how to setup these pages and you will be direct to go to the Google Webmasters tools site, and enter your sitemap and url information. read and sign up here.  https://www.google.com/webmasters/tools/home Free you need not pay unless you want ads.
Once you have this account you may also want to sign up for the adsense program, so you may place related Google ads to you site.

Please Note that you will also need a robots text, that will tell the spiders where not to index or craw pages and images etc.

So first you may want to check and see if you have a current robots text on your root, or search the term and find out how to create one. Or if you want I can help you with that in my next post, to come shortly!

I will follow up soon!

EDIT: please find an example below of a robots text:

Code Select Expand


# Allows only major search engines and known friendly spiders

# Major Search Engines and Known Friendly Spiders (allowed)

User-agent: Googlebot
Disallow:

User-agent: MSNBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: Teoma
Disallow:

User-agent: Twiceler
Disallow:

User-agent: Gigabot
Disallow:

User-agent: Scrubby
Disallow:

User-agent: Robozilla
Disallow:

user-agent: Yahoo

# Everyone Else (NOT allowed)

# Disallows cgi-bin and images,docs,Themes,mgal_data,sources/txt,dataUser-agent: *Disallow: /cgi-bin/Disallow: /images/Disallow: /docs/Disallow: /admin/Disallow: /Themes/Disallow: /mgal_data/Disallow: /sources/Disallow: /txt/Disallow: /data/



Should be named "robots.txt" using note pad!

regards,
Maxx

[Skhilled]

Thanks, Maxx.

[beast]

  Do I need one on my site?

[Skhilled]

Using a sitemap will help search engines know how to crawl your website. You can put things in it like what directories it is allowed or not allowed to crawl (index). It will help with your ranking as well.

Robots.txt will is more for telling them which robots are allowed or not allowed to crawl your website. Bad bots will not pay attention to this file and will do as they please. If you have problems with bad bots then use .htaccess to block them.

[nend]

Actually a sitemap doesn't tell the robot what to not index, it just tells it where content pages are. Also in your robots.txt you can list your sitemap in there. This makes it easier on new search engines to find it. Most search engines will not even look for a sitemap unless specified in the robots.txt. Some like Google you can specify it in webmaster tools, but not all so be sure to link your sitemap in your robots.txt.

Line in robots.txt to tell robots where your sitemap is.

Code Select Expand

Sitemap: http://my.site.com/sitemap.xml

Also note the robots.txt file, to allow all robots use this line. Then follow up by the dissallow lines for robots you don't want to index your site. Allow means go ahead, Disallow means don't look through my site.

Code Select Expand

User-agent: *
Allow: /

[Skhilled]

Thanks, bro. I planned on adding some things about what goes into a robots.txt file.

[Maxx]

Yes .... and I have also added a User/Visitor's Site map and Menu (Visual Tittle Links) both to my site using the TinyPortal Options to do this. that way you may list the most visited categories or sections of your site, for easy navigation for the visitors and spiders to find.  I have on my FrontPage the site map links and on the Forum page the Menu, basically the same, how it's up to you how to use these, in you TP controls you will find the option to create a menu with your site or the other other way around. I know other Portals also have this feature, but I'm using TP for an example, and it worked out just the way I wanted it to. Doc has a Menu here and it can be expanded has he knows, to include the Forums or article categories he'd like to direct his visitors to.... Another great thing that I'm not using but will be digging into is the use of a tagging systems, but it must have the option to credit direct relevant takes to the given subject, if that make and cents!

And please don't forget great content and it's relationship to your sites name and description and keywords ( Meta tags). There are many thing to learn daily, even for the ole like me!

And the Nice sites like this one are great places to start and/or continue.

regards,
Maxx