I have made a little patch. It creates the feature to exclude search engines and other bots from statistics.

CommentFileSizeAuthor
#20 interdiff-2433955-16-20.txt591 bytesricovandevin
#20 drupal-exclude-bots-in-statistics-2433955-20-D8.patch4.24 KBricovandevin
#16 interdiff-2433955-13-16.txt755 bytesricovandevin
#16 drupal-exclude-bots-in-statistics-2433955-16-D8.patch3.52 KBricovandevin
#13 drupal-exclude-bots-in-statistics-2433955-13-D8.patch3.32 KBricovandevin
drupal_7_34_statistics_exclude_bots.patch1.88 KBbrtamas
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mgifford’s picture

mgifford
he/him
Primary language English
CreditAttribution: mgifford commented

Nice. To get it in Core we'd have to bring it into D8 first so the functionality could be backported.

timmillwood’s picture

timmillwood
Primary language English
Location 🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales, UK
CreditAttribution: timmillwood at Appnovation for Pfizer, Inc. commented
Status: Needs review » Needs work

Great thought (and patch) but as @mgifford says, this needs to be against the 8.0.x branch and not against 7.34.

David_Rothstein’s picture

David_Rothstein CreditAttribution: David_Rothstein as a volunteer commented
Version: 7.34 » 8.0.x-dev
Issue tags: +Needs backport to D7
mgifford’s picture

mgifford
he/him
Primary language English
CreditAttribution: mgifford commented
Assigned: brtamas » Unassigned

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mallezie’s picture

mallezie
Primary language Dutch
Location Loenhout
CreditAttribution: mallezie at Wunder commented
Version: 8.1.x-dev » 8.2.x-dev

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Wim Leers’s picture

Wim Leers
Location Ghent 🇧🇪🇪🇺
CreditAttribution: Wim Leers as a volunteer commented
Related issues: +#2883776: /core/modules/statistics/statistics.php These pages use this blocked resource. If Googlebot can't access important resources on your page, the page might be indexed incorrectly.

Ironically, we also have #2883776: /core/modules/statistics/statistics.php These pages use this blocked resource. If Googlebot can't access important resources on your page, the page might be indexed incorrectly..

Which is true? What's the behavior we want?

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet for One Shoe commented

I guess the desired behavior is that the resource (/core/modules/statistics/statistics.php) is available to crawlers but it should not record the view.

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet commented
Status: Needs work » Needs review
FileSize
drupal-exclude-bots-in-statistics-2433955-13-D8.patch3.32 KB

A first patch that applies against 8.6.x-dev.

Status: Needs review » Needs work

The last submitted patch, 13: drupal-exclude-bots-in-statistics-2433955-13-D8.patch, failed testing. View results

timmillwood’s picture

timmillwood
Primary language English
Location 🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales, UK
CreditAttribution: timmillwood at Appnovation for Pfizer, Inc. commented 
+++ b/core/modules/statistics/statistics.php
@@ -16,10 +16,28 @@
+$userAgent = $container
+  ->get('request_stack')
+  ->getCurrentRequest()
+  ->headers
+  ->get('User-Agent', '');
+
+$views = $statisticsSettings->get('count_content_views');
+
+if ($statisticsSettings->get('count_content_views_exclude_bots') && $userAgent) {
+  $regexes = preg_split("/\r\n|\n|\r/", $statisticsSettings->get('count_content_views_exclude_bots_regex'));
+  // Do not record the view if the user agent string matches against one of the
+  // regexes.
+  foreach ($regexes as $regex) {
+    if (preg_match($regex, $userAgent)) {
+      $views = FALSE;
+      break;
+    }
+  }
+}

I'd fear that this could make the already slow statistics.php even slower.

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet commented
FileSize
drupal-exclude-bots-in-statistics-2433955-16-D8.patch3.52 KB
interdiff-2433955-13-16.txt755 bytes

It will undoubtedly make the script slower. If anyone has better ideas I'd be happy to turn those ideas into patches. For now let us work on fixing the issue.

Fixed the issue that the request is not pushed onto the request stack.

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet commented
Status: Needs work » Needs review

Forgot to change the issue status. :-$

Status: Needs review » Needs work

The last submitted patch, 16: drupal-exclude-bots-in-statistics-2433955-16-D8.patch, failed testing. View results

timmillwood’s picture

timmillwood
Primary language English
Location 🏴󠁧󠁢󠁷󠁬󠁳󠁿 Wales, UK
CreditAttribution: timmillwood at Appnovation for Pfizer, Inc. commented

I wonder if there's anything we can do at the JS level?

Failing tests are related to there not being a schema to match the new config settings, quite a simple fix.

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet for One Shoe commented
Status: Needs work » Needs review
FileSize
drupal-exclude-bots-in-statistics-2433955-20-D8.patch4.24 KB
interdiff-2433955-16-20.txt591 bytes

Fixed schema.

Wim Leers’s picture

Wim Leers
Location Ghent 🇧🇪🇪🇺
CreditAttribution: Wim Leers as a volunteer commented
Status: Needs review » Needs work

Thanks for working on this!

When #2502313: Installing the Statistics module doesn't do anything, one must also know about some pretty mysterious setting and enable it lands, we won't have any configuration left. This would bring back configuration. I'm also not convinced. Furthermore, that list of bots would need to be updated continuously. That's not workable.

ricovandevin’s picture

ricovandevin CreditAttribution: ricovandevin at Finlet for One Shoe commented

I guess we can safely assume that nobody wants to have bots polluting the statistics. So we can leave out configuration to enable the blocking of bots.

Furthermore, that list of bots would need to be updated continuously. That's not workable.

I think that is a good point. But do we have another mechanism for recognizing bots except for inspecting the User-Agent header?

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

quietone CreditAttribution: quietone at PreviousNext commented
Status: Needs work » Postponed

Statistics is approved for removal. See #3266457: [Policy] Deprecate Statistics module in D10 and move to contrib in D11

This is now Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.

It will be moved to a contributed Statistics project once the project is created and the Drupal 11 branch is open.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.