Search Engine Submission - Web Site Submission

SEO digest

It’s unfortunate that many SEO firms still offer “search engine submission” as part of their services. Most people in the SEO industry know there is little to no value in using search engine Submit URL pages, although Google has just updated its Google web site submission interface.

The major search engines (Ask, Google, Live, and Yahoo!) do a pretty good job of discovering new domains and crawling them. They are assisted by several domain name indexes that are continually updated as people register domains, and also by a few Wiki-style sites (such as Aboutus.org) that provide information on domains. Search engine submission is not quite a thing of the past, but it’s certainly not a service anyone needs to pay for.

People using free hosting sub-domains are not so lucky, although some free hosting providers create directories or drop links between sister sites to help their users’ content achieve search visibility. Still, if you create a free site it’s less likely to be found by the search engines quickly so you do want to exercise the available Web site submission options as soon as your sites are up and running.

Google, Live, and Yahoo! all provide Webmasters with the means of registering their sites with the search engines. Site registration is a much more satisfying experience than the old “submit your URL and we may crawl it” experience. So search engine submission has evolved more into a registration process for search engine optimization.

Google Webmaster Tools (listed in the web site submission interface linked to above) allows you to verify your access to the site, submit XML Sitemaps, and manage crawl rates and canonicalization. The linking data that Webmaster Tools provides is not useful and should be ignored (because it’s incomplete, it’s usually out of date, and it doesn’t distinguish between links that pass value and links that don’t pass value). Other data may be somewhat useful.

Yahoo! Site Explorer is not as robust as Google Webmaster Tools but you can verify or register your site with Yahoo!, tell Yahoo! where to find XML sitemaps, and monitor your progress. Yahoo! will also provide you with robust backlink reports — unfortunately, many of the links Yahoo! tells you about either don’t exist or don’t pass value, so why bother?

Live Webmaster Tools is a little more complicated than either Google or Yahoo!’s interface but it’s robust enough that I now feel it’s worth the effort to register sites with it. Like Google and Yahoo!, Live lets you verify your site and indicate where your XML sitemaps are.

XML sitemaps are controversial even though they have been around for a few years now. Some people swear by them. Some people swear AT them. Most people have experimented with them. There are, of course, some limitations with XML sitemaps. The search engines don’t seem to tolerate all line terminator formats, for example. That’s annoying. Perfectly good sitemap files are rejected because someone used the wrong text editor. All the search engines have to do is run a search-and-replace function on the files to normalize the line terminators.

One of the greater problems with XML sitemaps, however, is that they don’t guarantee that large sites will be fully crawled and indexed. Some people feel that breaking up their page lists into small sections and putting each section in its own sitemap works better. Some people feel that submitting multiple sitemaps works better. Since the search engines don’t guarantee every page will be crawled and indexed, they don’t object to seeing multiple sitemaps.

There are alternative formats for people who cannot for whatever reason use XML sitemaps. You can just create a text file that contains nothing but page URLs. Yahoo! insists that the file be named urllist.txt but Google only seems to care if the file uses a “.txt” extension (and I may be mistaken on that point, but that is my understanding of their requirement).

You can list your sitemaps (both XML and TXT) in your robots.txt file but for some reason you cannot link to them from your HTML pages (for discovery). That strikes me as rather silly but that’s just the way it is. On the other hand, you can link to RSS feeds from your HTML pages now, so why not include your page URLs in an RSS feed?

RSS feeds were originally created for Netscape’s XML Channels (and the acronym stood for Rich Site Summary, not Really Simple Syndication). You can still use RSS feed in that way. These old-school RSS feeds are used by some aggregator sites to create dynamic directories that index some pages on Web sites. Those aggregator sites may or may not help your site be crawled. (Remember, if a spammer can abuse a site, it probably HAS been abused.)

Most of the old aggregator sites are gone now but I still see a few around today.

Of course, you can always create your own HTML crawl pages. These can be user-friendly HTML Sitemaps or just plain ugly index pages that list lots of URLs. Search engines don’t seem to have any objections to crawl pages, but don’t waste your time using crawl pages as your only or primary source of links for pages you’re too ashamed to show to visitors.

If you’re in a hurry to get your pages indexed and ranking about the best thing you can do is create a blog on a free blogging site like Blogger, LiveJournal, Typepad, or Wordpress. Assuming you have sense enough not to dump 5,000 posts on a blog each day, you can write a post about each page on your site and link to the page with unique but descriptive (and SERP-helpful) anchor text.

These free blogs ping the blog search services (although you may have to turn on the pinging in your user control panels). The major search engines do crawl major blog search services.

You can also just put a blog on your Web site and use it in a similar way. As long as your blog pings the blog directories, you should be fine. The search engines will crawl your blog post and from there they should crawl the page you link to.

Although this is a slow method of getting indexed, it does help draw traffic to your site. And it may attract some links from other blogs.

Common mistakes to avoid if you decide to use blogs to help with search engine submission:

1. If one blog is good for me, 100 automated blogs will be great
2. I can use automated emails to create the blog posts
3. I can embed links to 20 pages in every post
4. I can replicate my pages on the blog to save time

Generally speaking, if you’re tempted to replicate your efforts or implement time-saving measures, you might as well brand “STUPID IDIOT SPAMMER” across your forehead now because your new site will soon be banned or penalized.

Avoid excess. Avoid following a pattern. Resist the temptation to be formulaic. Just because Dipstick Dave seemed to get away with it doesn’t mean you will.

Benefits of using blogs to help get your site crawled include:

1. Free blogging services often send visitors to random member blogs
2. Free blogging services provide you with a coveted “other domain/host” to link from
3. Free blogging services usually give you sub-domains, meaning you’re relatively isolated from other people’s stupid spamming tricks
4. Free blogging services handle software installation and basic configuration for you
5. Free blogging services usually have pretty good servers

If you need to get a lot of pages crawled and indexed quickly, there are some relatively risk-free methods for doing so. But they do require time, effort, and patience. Yes, some spammers can get large sites crawled very fast. But those sites rarely last long in the search indexes.

www.seo-theory.com

published @ September 30, 2008