/core/modules/statistics/statistics.php These pages use this blocked resource. If Googlebot can't access important resources on your page, the page might be indexed incorrectly.

Do you know for a fact this is blocking your site from being crawled? It seems an ordinary notice.

I have this warning with all 7.x stable as well (currently 7.56). Although statistics is enabled across all pages of my site and a lot or most of them look indexed by google, several (1400 of 27000) are listed as blocked because of statistics.php in "blocked resources" of Search console (aka Google webmaster tools).

Is the bug described in this thread the source of the constant:

"page not found" 

http://{site}/core/modules/statistics/statistics.php

warning/errors in the recent log messages for nearly every URL on any D8 site that enables core statistics?

***********

Do we need to specifically disallow the full path in robots.txt? we already have Disallow: /core/ so this should be covered, right?

#6: Isn't that exactly what's causing this? I think we need

Allow: /core/modules/statistics/statistics.php

Ironically, we also have #2433955: Statistics - Exclude bots (search engines, etc.).

Which is true? What's the behavior we want?

I guess the desired behavior is that the resource (/core/modules/statistics/statistics.php) is available to crawlers but it should not record the view.

The patch from #6 will work to solve the 'blocked resource' issue. We will need to block recording of the hits of crawlers to have more reliable statistics but that is worked on in #2433955: Statistics - Exclude bots (search engines, etc.).

Aligned version with release cycle.

Hmmm. Don't we need to do #2433955: Statistics - Exclude bots (search engines, etc.) first? At the moment hits are not recorded by bots because we block the resource right? Applied this will change that.

#7 will only work while you have not moved statistics.php elsewhere.

I wonder if the solution lies in X-Robots-Tag: https://developers.google.com/search/reference/robots_meta_tag

I don't see a solution in X-Robots-Tag (yet). As far as I can find out now there is no directive telling a bot to stop crawling a resource. You can tell a bot not to include the resource in a search index but that will not prevent the bot from requesting the resource (and thus recording views in case of /core/modules/statistics/statistics.php). But... I might be overlooking something. :-)

What is the correct approach to make sure that the core statistics module works again?

The statistics counter is frozen to an old value and won’t update (see comment #2883776-4: /core/modules/statistics/statistics.php These pages use this blocked resource. If Googlebot can't access important resources on your page, the page might be indexed incorrectly.). This fills the watchdog with errors.

Editing the robots.txt file to allow the not found URL does not repair the statistics.

Allow: /core/modules/statistics/statistics.php

I don't understand why we should allow crawlers to access this, they shouldn't.

This isn't related to fixing a frozen value, the value is supposed to update once per our but I suppose the page cache gets in the way as it doesn't respect that. But that has nothing to do with this issue.

I searched for an issue trying to fix the "page not found" watchdog error and found this issue.

Looks like both Drupal statistics module and Google cannot find the page (/core/modules/statistics/statistics.php) and update the statistics using the Statistics module.

If this is not related, I will open a new issue.

How do I apply this patch using composer, the code I use drupal/core I see an error error.

How to apply a patch to web root files?

"extra": {
        "composer-exit-on-patch-failure": true,
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "file-mapping": {
                "[web-root]/robots.txt": false
            },
			"locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": ["type:drupal-core"],
            "web/libraries/{$name}": ["type:drupal-library"],
            "web/modules/contrib/{$name}": ["type:drupal-module"],
            "web/profiles/contrib/{$name}": ["type:drupal-profile"],
            "web/themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/Commands/contrib/{$name}": ["type:drupal-drush"]
        },
        "patches": {
          "drupal/core": {
            "Fixed robot.txt": "https://www.drupal.org/files/issues/2883776-7.patch"
          }
        }
    }

Statistics is approved for removal. See #3266457: [Policy] Deprecate Statistics module in D10 and move to contrib in D11

This is now Postponed. The status is set according to two policies. The Remove a core extension and move it to a contributed project and the Extensions approved for removal policies.

It will be moved to a contributed Statistics project once the project is created and the Drupal 11 branch is open.