Google’s Supplemental Index: Get Out and Stay Out
Google is by far the most frequently used search engine, so naturally ranking well in Google is important. However, ranking well and maximizing your traffic requires getting and keeping your pages out of the supplemental index.
What is Google’s Supplemental Index?
Google’s goal as a search engine is to provide the user (searcher) with the most relevant search results possible. In order to produce the best search results, Goggle excludes certain pages from appearing in the SERPs (search engine results pages). The pages that are excluded are then thrown into what Google calls the supplemental index.
The supplemental index is just what it says, a supplement to the main index. Pages in the supplemental index may show up in search results, but pages in the main index will be given priority.
Why is it Important to Keep Your Pages Out of the Supplemental Index?
With the incredibly massive (and growing) number of searches done through Google, it’s not uncommon for websites to get 60% or more of their traffic from Google. Obviously, if you have more pages from your site being indexed and displayed by Google, you’re likely to get more traffic. Decreasing the number of pages that are in Google’s supplemental index can significantly increase your traffic virtually overnight (see Nathan Metzger’s article about how he increased his traffic by 20%).
Why Do Pages Get Put Into the Supplemental Index?
There are a number of reasons that a page can get thrown into the supplemental index. The most common reasons (and more importantly, solutions) are discussed below.
1 – Duplicate content
The most common cause of a page being condemned to the supplemental index is duplicate content. Google wants the pages in its search results to be unique. Searchers are likely to become frustrated if the information they find through a search is nothing more than the same exact content on different pages and sites. In order to avoid this, Google eliminates pages with duplicate content from its search results by keeping them in the supplemental index.
Duplicate content issues are especially common with blogs. A page on a blog that contains a blog post is duplicated when the post reappears on the homepage, category pages, and date pages. You can get around this by:
– Displaying only excerpts of the posts on the homepage, category pages, and date pages. This shows the search engine only a portion of the post, which is usually not enough to have it flagged as duplicate content.
– Use a robots.txt file to tell search engines to ignore parts of the blog/site that are duplicated. This will help to make sure that the original post page is not caught in the supplemental index. For more information on your blog’s robots.txt, see Not So Boring Life.
– The Duplicate Content Cure plugin is another option for WordPress users. This plugin tells the search engines to ignore the pages that contain duplicate content. By default the plugin prevents search engines from indexing category pages, but that can be changed (which I recommend doing). Category pages can contain strong, related keywords that will produce good search engine rankings.
2 – Lack of text on a page
Pages with little text/content are more likely to be placed in the supplemental index. Pages that only contain links are also likely targets.
Add more valuable content (not just links) to any important pages that are in the supplemental index.
3 – Long URLs
Pages with longer URLs and with more variable characters (question marks, hyphens, etc.) are frequently placed in the supplemental index. This is not always the case. Plenty of pages that rank well in Google (especially pages on a blog) have long URLs. The long URL is not always a problem, but when combined with some of the other items on this list it is more likely to cause Google to question the page.
For static websites you can give your files shorter names or decrease the number of folders that are used, which can also add to the length of the URL. WordPress users have the option to specify the URL structure that should be used. From your dashboard go to “Options and Permalinks”.
4 – Repetitive page titles
Every page on your website should have a unique title. Simply using the company name or website name for the title on every page does not help the search engines to determine the subject/topic of each individual page.
Give each page a unique and descriptive title. WordPress users can download the All In One SEO Pack plugin that will allow you to enter the page title as well as the meta tags for each post.
5 – Orphaned pages
An orphaned page is a page on your site that has no links from other pages on your website. If no other pages link to it, Google assumes it is not important or it has no valuable information.
Every page that you hope to have indexed by search engines should have links pointing towards it. One way of accomplishing this is to use a sitemap that includes a link to every page. A sitemap alone is unlikely to make enough of a difference, though. Use links throughout the site wherever they are relevant.
If you have no links to a particular page because you don’t want it to be found, include it in your robots.txt file.
6 – Buried pages
Google takes into consideration how close or far a page is from the site’s homepage (meaning how many clicks from the homepage it takes to reach a certain page). As a general rule of thumb, all pages on your site should be able to be reached from the homepage in two clicks or less (not always possible on larger sites).
– The key is to improve the links to a page throughout the site. Look for relevant pages that can link together. This is the most powerful way to show search engines that the page is important.
– Use links in the text of articles, as well as at the end of articles (to related posts)
– Link to your most important pages from your homepage.
– The sitemap, mentioned under “Buried Pages”, also helps here. Link to the sitemap from your homepage and every page on the sitemap will be only 2 clicks from the homepage.
How can I see which pages are in the supplemental index?
There used to be an easy search to see the pages in the supplemental index, but a few weeks ago Google disabled the search. For now the best way that I know of is type the following in the Google search box: “site:www.yourdomain.com”.
When the search results come up you will see all of your pages that are indexed by Google. This includes pages that are in the main index as well as the supplemental index. The pages from the main index will be listed first. The pages from the supplemental index will say “supplemental result” in green text next to the URL. You will need to click through the pages to find the point where the pages from the main index end and pages from the supplemental index begin. [UPDATE: GOOGLE NO LONGER DISPLAYS WHICH LINKS ARE IN THE SUPPLEMENTAL INDEX]
From my recent personal experience with this site and with a client’s site the robots.txt file, post excerpts, and internal linking have made a huge difference (getting some inbound links from external sites has helped as well). On this particular site we have increased the number of pages in the main index by over 100 in the past 2 or 3 weeks.
Try some of the methods above and you should be able to get your most important pages in the main index and bring in the traffic.