Bot Logger

Latest: bot-logger-1.7 (New) | Author: Mike Ciffone

Version 1.7 adds backward compatibility for systems running PHP 7.4.

Bot Logger is a WordPress plugin designed to log and validate incoming requests from search engine bots. It logs Googlebot and Bingbot by default and allows up to 3 custom user agents (for now). Bot Logger was designed to make log file analysis more convenient and accessible. It enables all SEOs, technical or not, to view details that are otherwise only shown in access.log files such as user agent, request date, resource, status code, and IP address directly in the WordPress admin panel.

Bot Logger's main log viewer page. The menu includes default tabs for Googlebot and Bingbot log information followed by a tab with a plus sign to add a custom user agent. Along the right are buttons to export data and validate IP addresses. The main table shows the latest requests from Googlebot including the identifying part of it's user agent string (Googlebot/2.1), the date stamp, the HTTP status code, the resource fetched, the source IP address, and the "IP Status" which are all showing "valid".

Motivation

When log file analysis is inconvenient, it doesn’t get done. As an SEO consultant, over the years I’ve realized that the true value of log file analysis is the instincts you gain about a site by regularly observing how search engines crawl it.

Truly understanding crawl behavior allows you to better internally link, predict ranking changes, understand what search engines think your best content is, and make more ROI driven decisions.

However, to gain that level of instinct required you have to be paying attention every day, and that’s why I built Bot Logger.

Features

  • Logs requests from Googlebot and Bingbot by default
    • Handles a variety of Google’s alternate user agents
  • Ability to import existing access.log data from the server (assuming www-data user has read access)
    • Bot Logger automatically looks in standard access.log paths (eg. /var/log/nginx/) for Nginx, Apache, and Litespeed. Node.js is not supported at this time.
    • If access logs are not in standard directories, users can set a custom path in the import page settings.
  • Automatic and manual IP validation against Google’s and Bing’s IP ranges
  • Ability to track custom user agents (eg. additional search engines, social media apps, scrapers etc)
  • Tabbed interface for viewing logs from different bots
  • Manage retention period and validation frequency in the settings page, as well as manual clearing of log data
  • Cloudflare integration automatically creates a cache-rule for custom user agents to ensure they hit the origin server and are logged

View the repository on GitHub.

Screenshots

 

Get in touch

I get a lot of spam, if a day goes by and you haven't heard back, message me on LinkedIn.

  • This field is for validation purposes and should be left unchanged.