Search Engine Optimization gone amok (Jim Carson) September 5, 2005

Categories

Search Engine Optimization gone amok

Since September 3rd, a bot at Yahoo (68.142.195.81) has been repeatedly, as in 45,266 times and still counting, polling my web site. I've finally added the magic line in .htaccess, but it's annoying that there's no way to contact a human being at yahoo to kick this thing in the pants. Their bot is requesting pages from my site every five seconds, ignoring the 10-second interval specified in my robots.txt file, and oscillating among three pages. Repeat this a gazillion times:

68.142.195.81 - - [05/Sep/2005:13:09:15 -0700] "GET /a/2005/08/raw_2005_wow.shtml HTTP/1.0" 403 - "-" "YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)" 68.142.195.81 - - [05/Sep/2005:13:09:20 -0700] "GET /a/2005/09/post_your_first_1.shtml HTTP/1.0" 403 - "-" "YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)" 68.142.195.81 - - [05/Sep/2005:13:09:25 -0700] "GET /a/2005/08/ride_around_was_1.shtml HTTP/1.0" 403 - "-" "YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)"

Testing, indeed. Irony alert! - after I posted this, the search bot soon changed its behavior to indexing this very blog entry over and over and over. At least it's doing it once every 11 seconds.

Posted by Jim on September 5, 2005 | Permalink

6 Comments:

jim wrote on (September 5, 2005 3:55 PM)

Ironically, now that I've posted this, the search bot has throttled itself back to once every 11 seconds, polling this particular entry. I wouldn't mind this so much if it netted me some kind of uber placement in the arcane topics in which I write.

The number associated with Yahoo's IP address goes to their HR benefits group, not the "network abuse" group, but did have a message referring me to the secret 408/349-1572 number for a live customer support person. Unfortunately, she didn't have any more insight.

Sam wrote on (September 5, 2005 10:20 PM)

Can your ISP just drop all packets with that source IP address, or do you care about Yahoo picking up your entries?

Terri wrote on (September 6, 2005 3:22 PM)

Wow, you're being stalked by robots? I think that I'm being stalked by rats. No seriously, did anyone ever recommend your site to Yahoo? How does all of this begin?

John Calnan wrote on (September 7, 2005 5:51 AM)

I've been stalked by rats. You are much better off being stalked by robots. If you turn on the sprinklers, they'll rust up and stop like in The Wizard of Oz.

jim wrote on (September 7, 2005 8:12 AM)

Sprinklers! I turned them on last night and noticed a slowing of the robots.

By the way, before my most recent robot infestation, I had a rat problem. Rats eat everything except bandwidth. Robots mop up the rest.

I'm not saying one leads to another, but you might want to water your lawn.

Just saying.

John Calnan wrote on (September 10, 2005 11:19 AM)

Not sure if this will help, but I ran across this spec on W3c today. Include the angle brackets, of course.

meta name="ROBOTS" content="INDEX,FOLLOW"
meta name="REVISIT-AFTER" content="30 days"

Sun	Mon	Tue	Wed	Thu	Fri	Sat
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31

Jim Carson

This is the older blog.

Categories