Some barriers to being indexed are due to the misbehavior of a handful of webmasters who have tried to fool search engines into ranking their pages high on lists of matches and including them as matches to queries they aren't appropriate for. This is one kind of behavior that is known as "spamming." Spamming degrades the value of the index and is a nuisance for all.

The logic that leads people to try such tricks is rather bizarre. "I figure everybody searches for the word 'sex.' I don't have any sex at my site, but I want people to stumble across my site. So I'm going to put the word 'sex,' three thousand times as comments. And any time that anybody searches for 'sex,' my pages will show up first."

People have actually tried that. They have tried doing the same kind of thing in the backgrounds of their Web pages. They have also created page after page of text that is in the same color as the background color so visitors won't see the words, but search engine crawlers will. They have tried everything imaginable to fool search engines.

In order to make the index cleaner and more navigable, and to foster a more level playing field for everyone, search engines have started removing these pages from the index and screening new submissions. If you load pages with long, repetitive word lists, or titles this will cause most engines either to ignore the repetition or, in some cases, to ignore such documents entirely.