If Your Website Is Slow, Consider Blocking Specific Bots

[Zwielicht]

I've already posted about this on the Xenforo forums and my business website, but I'm posting it here as well as many of you many find this information useful. The bots I talk about don't just visit forums, but websites generally.

Before I continue, you can see how I syndicated the content here. I rewrite everything so that's it's not exactly the same rather than copy/paste it as is. Feel free to do the same here on Office Outlaw to grow your business sites!

Anyway, if you've kept up with the changelog here on Office Outlaw, you'll know that I made some changes yesterday to deal with the page load speed issue. As the forum has grown since March, we've been getting more bots visiting the site. At first, these bots we wanted like Googlebot and Bingbot, but as time went on, we started getting more undesirable bots that offered absolutely no benefit to us. In fact, these bots were visiting the forum so often, sometimes hundreds of times in a very short span of time, thus slowing down the website as it was like over a thousand people loading a page at once.

Frustrating, right? The worst part was that upon further research, some of these bots reportedly ignored robots.txt files. I ended up blocking them via Cloudflare, although if you don't use Cloudflare, you can block them via .htaccess or even a bot management tool if your web hosts has one like mine does.

Blocking via .htaccess example

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} Rogerbot [NC]
    RewriteRule ^ - [F,L]
</IfModule>

Blocking via robots.txt example

User-agent: Moz
Disallow: /

If you're curious to know which bots I blocked and the reasons why, I've provided a list below:

• Bytespider: By far the worst one. I found out many other Xenforo forum owners were dealing with this one swarming their forums and ignoring robots.txt files. It’s Bytedance AKA TikTok’s bot that trains their language learning models, which means there’s absolutely no benefit to you for allowing this one to crawl your website.
• Amazonbot: You can consider allowing this one if you’ve found people use Alexa with your site and you want to keep that feature working.
• Semrush: This one is useful if you’re using tool or if you plan on selling your website in the future as many in the industry feel they’re fairly accurate with collecting data. Since I don’t use the tool and I don’t plan on selling my forum, I’ve blocked it.
• Ahrefs: Same situation as before where it's only useful if you're using the tool.
• Mozlinkexplorer: Again, only useful if you're using the tool or plan to use it in the future.

 

[SilverClouds]

Blocking these AI bots especially is such a good idea, they basically come to rip your content . Thanks, I'll look into it.

 

[roydan]

Amazonbot: You can consider allowing this one if you’ve found people use Alexa with your site and you want to keep that feature working.

Wasn't Alexa killed a few years back?

Blocking these AI bots especially is such a good idea, they basically come to rip your content . Thanks, I'll look into it.

I wouldn't. They maybe rip your content but at some point they'll be all that users use, so you're either there or you're potentially staying behind the competition.
If you run a proper business site (agency, stores, etc) you should not only let them in, but to optimize your sire for them and serve them coffee when they show up.
For blogs I'm not entirely sure yet, I let them in, still, in my blogs.

 

[t2van]

Why not adjust your server cache settings / play around with those settings, I can't see a robot doing much to effect the site speed to be honest..

Adjust the size of your tmp file and how often it's cleaned out.

Adjust my.cnf to improve the speed of things there are some good resources online I found a gem of an IBM one years ago I'll have to try and locate it again.

Optimizing my.cnf for MySQL performance

Database systems often need tuning for best performance. Lukas Vileikis explains to how to optimize my.cnf for MySQL performance.

www.red-gate.com

That one will do ^

If those things are not an option due to hosting, check if gzip, expires, deflate (mod_gzip, mod_expires, mod_deflate) all installed on the hosting and try playing with those settings

 

[Zwielicht]

Why not adjust your server cache settings / play around with those settings, I can't see a robot doing much to effect the site speed to be honest..

Adjust the size of your tmp file and how often it's cleaned out.

Adjust my.cnf to improve the speed of things there are some good resources online I found a gem of an IBM one years ago I'll have to try and locate it again.

Optimizing my.cnf for MySQL performance

Database systems often need tuning for best performance. Lukas Vileikis explains to how to optimize my.cnf for MySQL performance.

www.red-gate.com

That one will do ^

If those things are not an option due to hosting, check if gzip, expires, deflate (mod_gzip, mod_expires, mod_deflate) all installed on the hosting and try playing with those settings

That's some great advice. I'll take a look at that. Thanks, @t2van.

Wasn't Alexa killed a few years back?

It's still around as Alexa+. They use it for the Amazon Echo from what I remember.

 

[29dogs]

Amazonbot: You can consider allowing this one if you’ve found people use Alexa with your site and you want to keep that feature working.

Does people using Alexa and searching your site with voice give you any benefit though?

Seems like it only benefits Amazon when they do that.

 

[Zwielicht]

Does people using Alexa and searching your site with voice give you any benefit though?

Seems like it only benefits Amazon when they do that.

From what I know, I don't think there's any direct benefit, but I could be wrong about this.