The aim of a search engine optimization program is to get your pages indexed by the search engines as quickly as possible. Once they have been indexed they can start to climb the search engine results page ladder. For some webmasters, getting their pages indexed is not always the problem, it is keeping them in the index that is a problem.
Former Googler Vanessa Fox has some tips on this topic published in both text and video format on WebProNews. Her tips and comments make interesting viewing/reading and make a lot of sense. Her tips and comments, in a nutshell are:
A traffic problem is not necessarily a ranking problem – it could be a crawl problem
This is true, although if your pages are not being crawled and indexed then they are not being ranked. However, her point is that a drop in traffic may be a case of losing search rank, it could be your pages have dropped out of the index altogether. There are any number of crawl issues, but one that she hasn’t covered is that of access – if your server is down too often then your pages will drop out of the index.
Organize your pages into categories
A well planned site should have its pages organized into categories. However, Ms Fox goes one step further and suggests you analyze your site on a category basis rather than on a page basis. This makes a lot of sense. If you have one or two categories that are working well and one that isn’t, you can either work harder on the category that isn’t working for you, or drop it altogether.
Manage the search engines crawl efficiency
Search engines won’t spend all day crawling your whole site. Help them out by using appropriate links to your most important pages. You can also use the nofollow attribute to stop the search engines from crawling unnecessary pages. When the search engine visits, get it to crawl the best pages first.
Create comprehensive canonical .xml sitemaps for each categories
This is interesting. Having a sitemap for each category can certainly help the search engines crawl your site. By creating a general sitemap index file you can tell the search engines which categories have their own sitemap and where it is. You can also use Webmaster Tools to assess each category.
Make sure you have a problem before you start to make repairs
This is the least thought of tip when people try to assess what has gone wrong. You can search everywhere for problems, apply tweaks here and there, make changes and generally go gray looking for a problem when all along it is just a simple crawl problem. Don’t make assumptions, check the above points first.
These are not the only areas to look at, and search engine crawling is only one part of a bigger picture. If your pages are not ranking well, if at all, the you need to look at a wide range of variables. This checklist is a good place to start if you’re trying to determine why your pages are not being indexed in the search engines.