- Compatible XF versions
- 2.2
- 2.3
This XenForo 2.x addon adds additional definitions for bot detection in sessions.
This allows more accurate indication of the number of users vs robots online at any given time.
New bot definitions are pulled automatically via API and potential new bots detected are sent back to the addon author for analysis for inclusion in the bot definitions.
As of October 2023, the addon can detect over 1,400 different bots and the API database now contains more than 3,200 user agent strings that have been identified as bots.
Statistics from the KnownBots API server management system, custom built to track and identify bots and serve the updates API:
Usage
When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page http://www.example.com/community/online/?type=robot
We also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.
There is a tool in the admin area for displaying a complete list of currently known bots:
... and another tool which lets you test a user agent string to see if it would be detected as a bot or not:
New bot definitions via API
In v4.x, by default this addon will query our KnownBots API for updates and automatically apply them - no more manually installing updates to get new bot lists. You may query the API directly at any time, and you may also disable the automatic update process in favour of manually downloading the json file and
loading the data in using a CLI script. Instructions for doing this can be found in the addon FAQ section.
Generic bot detection
User Agent strings are scanned for the keywords "bot", "crawl", or "spider" - any User Agents not already detected as a bot which contain one of these strings are stored in the cache and made visible through the admin UI, with the option to have this information emailed on a weekly basis (not enabled by default).
The default email address to send new bots to will be delivered to the addon developer - however, the admin is free to have this information sent to any address they choose and can manually forward the detected bots to me.
To turn on automatic emails of detected user agents, check the Email New Bots option and enter an email (or leave it at the default setting).
There is an admin UI tool to display recently detected bots with a button to send the bot list email immediately:
Privacy Statement
This allows more accurate indication of the number of users vs robots online at any given time.
New bot definitions are pulled automatically via API and potential new bots detected are sent back to the addon author for analysis for inclusion in the bot definitions.
As of October 2023, the addon can detect over 1,400 different bots and the API database now contains more than 3,200 user agent strings that have been identified as bots.
Statistics from the KnownBots API server management system, custom built to track and identify bots and serve the updates API:
Usage
When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page http://www.example.com/community/online/?type=robot
We also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.
There is a tool in the admin area for displaying a complete list of currently known bots:
... and another tool which lets you test a user agent string to see if it would be detected as a bot or not:
New bot definitions via API
In v4.x, by default this addon will query our KnownBots API for updates and automatically apply them - no more manually installing updates to get new bot lists. You may query the API directly at any time, and you may also disable the automatic update process in favour of manually downloading the json file and
loading the data in using a CLI script. Instructions for doing this can be found in the addon FAQ section.
Generic bot detection
User Agent strings are scanned for the keywords "bot", "crawl", or "spider" - any User Agents not already detected as a bot which contain one of these strings are stored in the cache and made visible through the admin UI, with the option to have this information emailed on a weekly basis (not enabled by default).
The default email address to send new bots to will be delivered to the addon developer - however, the admin is free to have this information sent to any address they choose and can manually forward the detected bots to me.
To turn on automatic emails of detected user agents, check the Email New Bots option and enter an email (or leave it at the default setting).
There is an admin UI tool to display recently detected bots with a button to send the bot list email immediately:
Privacy Statement
- Since v3, the addon has included functionality to email the list of new bots detected automatically by a forum to an email address configurable by the admin.
- The default and recommended value is to email "knownbots at hampel.io"
- Emails sent to this address will only be used for the purposes of identifying new bots to add to this addon
- Email addresses will NOT be sold or added to any marketing lists - not even ours.
- If there are issues detected from the emails you are sending us, we may reply to establish communication - but that will be on a case-by-case basis and only for the purposes of troubleshooting the operation of this addon.
- You can check the information contained in the emails by changing the address temporarily to your own so that emails go to you. If you still want us to process these emails - please feel free to forward them to the above address.
- Other than a list of user agent strings, the only information contained in the email will be those automatically added to the email header by the forum mailer and SMTP servers.
