Comments on: What is the best way to deal with Spiders/Bots/Crawlers?

By: secure vpn

secure vpn — Fri, 27 Aug 2010 14:57:18 +0000

Usually you should know if you’re connecting to a VPN, because you have to specifically set up a VPN. It’s not something that happens when you go to a website, its something you configure intentionally. A secure https connection is not a VPN connection.

By: erik

erik — Mon, 03 May 2010 23:45:28 +0000

Does someone have some good code examples of this to share?, or maybe a .cfc? Would be greatly appreciated.

Erik

By: car magnets

car magnets — Mon, 05 Apr 2010 02:48:27 +0000

I might keep this handy for clients as well as for myself. Have a great Easter!

By: Anuj Gakhar

Anuj Gakhar — Mon, 08 Feb 2010 08:03:34 +0000

Michael, since you have something like this ready already, you mind sharing it here?

By: Erik

Erik — Wed, 03 Feb 2010 18:52:43 +0000

Michael, Anuj…

Could someone post example code that does what Michael suggests in an App.cfc format? Thanks!

By: Anuj Gakhar

Anuj Gakhar — Fri, 29 Jan 2010 10:00:33 +0000

Hi Michael, Thanks for the explanation. It does appear to be an interesting technique, One that could solve a lot of problems around this “bot” issue. Let me know when you update your blog post on the subject.

By: Michael Dinowitz

Michael Dinowitz — Fri, 29 Jan 2010 04:17:21 +0000

@ziggy
I create a structure which will use the visitor’s IP address as a key and the number of times they’ve been to the site that second as the value for that key. If the value is greater than 3, I then abort that request. I clear out the structure in the next second. A bit of memory hashing, but not anything major in the least.

@Anuj
Yes, every bot has a cfid/cftoken of 1 so they all have the same session. This avoids the problem of each bot getting a new session, even if the bot’s session timed out after 2 seconds (my previous method). Every time a bot gets it’s own session, it incremented the cfid counter and forced CF to generate a cftoken for it. Again, not any real performance hit but why have the work done if not needed.

It is rare for a legitimate bot to visit a page more than once or twice a second. I’ve found that certain firewall products will spider a site with a dozen or more requests a second. This can be very detrimental as the same spider has some really strange connection delay which holds the connection open for a while rather than getting the page results immediately. So the obvious result is a dozen CF connections, all hung waiting to deliver data and a few dozen more being added to the queue every second or so. Can you say crash? I can.

I’ve got to update my blog post on this sometime.

By: Robert Zehnder

Robert Zehnder — Thu, 28 Jan 2010 15:47:52 +0000

I have a very basic list of bot user agents that I use in WhosOnCFC. If you would like a starting point you can find the current active list at http://www.kisdigital.com/botlist.xml. I do a findNoCase against the user agent with the list and it does a decent job of finding the bots that advertise themselves.

By: Anuj Gakhar

Anuj Gakhar — Thu, 28 Jan 2010 13:24:14 +0000

That sounds pretty promising, Michael. So you are saying, every bot shares the same session no matter which bot it is. googlebot or yahoo slurp etc. But normally bots visit tens of pages within a second – so you dont let them visit more than 3 (or whatver the threshold is) pages per second ?

By: ziggy

ziggy — Thu, 28 Jan 2010 05:08:48 +0000

How do you detect how many connections they have?