East of the Sun, West of the Moon

2005/11/18

Who or what?

Filed under: Software — Erwin @ 2:05 pm

Does anyone reading here know who or what RufusBot (Rufus Web Miner) is and what the big deal is? Most search-engine bots check a few pages and come back another day to check some more or verify ones they looked at before. This one seems to just fetch the entire website in one go (dgd.is-here.com in this case) and it feels a bit excessive.

I’ve googled for it but all I seem to find is other webmasters wondering what’s up, so that’s no help. Yes, I realize I’m adding more to that noise, but…

Powered by WordPress