Would you like to learn how to configure the Apache server to deny access to bad Bots and Crawlers? In this tutorial, we are going to configure the Apache server to block the access from bad Bots and Crawlers.
• Ubuntu 20
• Ubuntu 19
• Ubuntu 18
• Apache 2.4.41
In our example, the Apache server is hosting the website WWW.GAMEKING.TIPS.
Copyright © 2018-2021 by Techexpert.tips.
All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means without the prior written permission of the publisher.
Equipment list
The following section presents the list of equipment used to create this tutorial.
As an Amazon Associate, I earn from qualifying purchases.
Apache – Related Tutorial:
On this page, we offer quick access to a list of tutorials related to Apache.
Tutorial Apache – Blocking bad Bots and Crawlers
Search the Apache log file for a list of User-agents.
A list of suspect User-agents that accessed your website will be displayed.
Create a list of User-agents to block.
Optionally, this GitHub project offers a list of bad Bots and Crawlers.
Enable the required Apache modules.
Edit the Apache configuration file for the default website.
Add the following lines to this configuration file.
Change the USER-AGENT values to reflect your needs.
Here is the file, before our configuration.
Here is the file, after our configuration.
Restart the Apache service.
In our example, the Apache server will forbid access to a list of Bots and Crawlers selected by the administrator.
From a remote Linux computer, test your configuration.
Here is the command output.
The Apache server will forbid access from specific USER-AGENT values.
From a remote Linux computer, try to perform access using any other USER-AGENT value.
Here is the command output.
The Apache server will allow any other USER-AGENT value to access your website.
Congratulation! You learned how to configure the Apache server to deny access to bad Bots and Crawlers.
