Comments on: Make a web spider http://www.scienceforseo.com/tutorials/make-a-web-spider/ a bridge between worlds Sat, 05 Feb 2011 14:02:08 +0000 hourly 1 By: phaithful http://www.scienceforseo.com/tutorials/make-a-web-spider/comment-page-1/#comment-737 phaithful Wed, 20 May 2009 01:31:32 +0000 http://www.scienceforseo.com/?p=945#comment-737 CJ, that's a great primer on how search engine spiders work and where to start if you want to build your own. I have to say that I learned quite a bit about technical SEO after I wrote my first spider. It worked great for valid HTML, but failed miserably to even the smallest malformed HTML comments. The most basic spider is simple to write, but dealing with all those errors, malformed code, and spider traps ... that's what makes up a large part of the code base. It's been a few years since I wrote my first spider, and I have to say that I've been very particular on writing very clean HTML and JavaScript since. Whether or not that has an impact on search rankings remains to be seen, but at least I've dreadfully aware of the potential spider traps when developing new sites. If you're a technical SEO, I'd highly recommend dabbling with spiders, just to gain some additional perspective.<!-- Touched by JuLiA --> CJ, that’s a great primer on how search engine spiders work and where to start if you want to build your own.

I have to say that I learned quite a bit about technical SEO after I wrote my first spider. It worked great for valid HTML, but failed miserably to even the smallest malformed HTML comments.

The most basic spider is simple to write, but dealing with all those errors, malformed code, and spider traps … that’s what makes up a large part of the code base.

It’s been a few years since I wrote my first spider, and I have to say that I’ve been very particular on writing very clean HTML and JavaScript since. Whether or not that has an impact on search rankings remains to be seen, but at least I’ve dreadfully aware of the potential spider traps when developing new sites.

If you’re a technical SEO, I’d highly recommend dabbling with spiders, just to gain some additional perspective.

]]>