index.php randomly appears in friendly URLs

Would you please reference the links you researched in the issue summary?

I have a feeling there is already a core issue for this but admittedly, I can’t find it.

I'm facing the issue with
D 8.8.5

I have nginx as reverse proxy. Posted about the issue with server configuration https://www.drupal.org/forum/support/post-installation/2020-04-06/solved...

Configuration:
nginx version: nginx/1.16.1
Server version: Apache/2.4.6 (CentOS)
PHP 7.2.29 with fast_cgi

Contributed module:

Adaptive themes
Admin Toolbar
Ctools
FAQ
Pahauto
Token
Metatag
Schema
AMP
Honeypot
XMLSitemap

Even I have tested with fresh D8 installation without an contributed module, it adds index.php in url. IE: mysite.tld/index.php/contact

I had this problem also.

Disabling the Sitemap module worked for me.

See this issue on the possibility of an interaction between the sitemap module and this problem:

https://www.drupal.org/project/sitemap/issues/3145126

The same issue happened to me before updating the simple_sitemap module to 3.7.0

We are currently seeing this intermittently on a production host. Google search console showed us duplicate content, alerting us to this issue.
This particular site has alpha, beta, stage, and production instances with the same code and a database restore capability where we have been able to confirm that the issue is intermittent. Cache clear and load of pages typically generates clean urls, but on occasion we see the /index.php/ injected at the start of the URL.

Confirming the contents of the problem in the cache:

mysql> select count(*) from cache_page where data like '%/index.php/%';
mysql> select count(*) from cache_render where data like '%/index.php/%';

Checking the cache_page table, I can confirm that Drupal has cached many pages with /index.php/. Additionally, checking the cache_render, there are cached instances of menus with /index.php/. Additionally, twig calls to url('') return the /index.php/ when this occurs.

Running a clear cache, visiting the pages to prime the cache, I now see zero result and url('') no longer returns the /index.php/. There have been no code deployments, nor module changes between runs where the /index.php/ was included vs. ones where it was not.

Further research is necessary.

We also facing the problem that 'index.php' appears randomly in the menus on two D8 websites on completely different servers, but not on dozens of other D8 instances. The 'index.php' URLs caused problems with the facet search, among other things.

It is particularly confusing that one browser (Anon) outputs index.php in the menus, while another browser displays the URLs correctly.

It seams that this issue appears more often when a new browser session has been started.

We activated the redirect module with 'Redirect from non-canonical URLs to the canonical URLs.' enabled, what redirects the 'index.php' paths to it's correct URL. But in the Menus 'index.php' still appears from time to time.

I had the same problem, and I didn't find the solution...

So I write this simple jquery code to be ok... (I know that this is a hack, not the official solution but... finally works)

 /* remove the weird "index.php/" from href's */
    var aHref = 0;
    var aHrefClean = 0;
    $('a[href*="index.php/"]').each(function(){
      aHref = $(this).attr('href');
      aHrefClean = aHref.replace("index.php/", "");
      $(this).attr('href', aHrefClean);
    });

Hi,

I found a couple of solutions for me on Platform.sh.

Nginx redirect
Updating my `routes.yml` to have the following redirect rule:

"https://www.{default}/":
  type: upstream
  upstream: "app:http"
  redirects:
    paths:
      '^(.*/)index\.php/(.*)':
        to: https://{default}/$2
        regexp: true
        code: 301

Redirects module
Checking the ' Enforce clean and canonical URLs.' checkbox in the settings had the same affect as above.

Hope this helps someone.

Cheers

Tom

I had a client bring the issue as described in #12 to our attention today. Even more puzzling is that in this particular site, all users are logged in. We both browsed the same page at the same time. When I inspected the element, the href was a normal relative clean path. Her href started with /index.php/ before the clean path. Our client was in Windows and could see this in both her browsers, as could our QA lead, also in a Windows browser. The same twig code on another page did not render the mungled href. As reported in this thread, a cache clear cleaned up the page.

It felt like an old episode of Twilight Zone until a colleague found this issue in the issue queue.

Also present on 9.1.x

Hi,

I confirme the "randomly aspect" of the bug.

mysql> select count(*) from cache_page where data like '%/index.php/%';
mysql> select count(*) from cache_render where data like '%/index.php/%';

request confirm the presence of links with "index.php" in URL in caches.

Cleaning cache from back-office change nothing, clearing with "drush cr" works.

But after some time (user trafic ? robot indexation ? ) some URL are cached with index.php.

Edit :
- It seems this affecte only the production (online-public) but not non-production sites (online-private => htaccess & co)
- No reverse proxy on stack (amazon AWS)

More ideas,

on my project, we use robotstxt module instead of "nativ robots.txt file".
Since now, we have not modification between native robots.txt file (from another project) and the version provide by the module.
The change are call to robots.txt url are let an entry on cache_page table.
Adding the fact that the robots.txt file contains indication for some "index.php" files (see below)

so when executing sql request
mysql> select cid, created from cache_page where data like '%/index.php/%' or cid like '%/index.php/%' order by created asc;
Note : cache_page "cid" are the URL called.

in results :
- the call to robot.txt seems call first.
- After came some page with "correct URL" containing some "prefixed" url on data
- And after "cid" containing prefixed url.

I think that some crawlers may generate somme buggy cached page...

robots.txt file :

[...]
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password/
Disallow: /index.php/user/register/
Disallow: /index.php/user/login/
Disallow: /index.php/user/logout/
[...]

Note : sitemap no containing bad url ...

#5 works for me. Enable Redirect module and in Redirect Module configuration enable "Enforce clean and canonical URLs". Clear varnish cache

Got the same problem with D9, 5# works for me as well.

I have D9.1.2.
From the defined homepage node all normal menu-links (with node reference) have /index.php/normallink.

If I go to a menu-link with an explicit link like "/news" (no reference to node id), I land on this page /news.
From this site /news the other links looks normal without index.php.

So if I am on index.php/xxx links seems correct. If I am on /news it seems that all links have index.php/xxx in the url

I solved it with clearing the complete drupal cache

I found Google had been indexing many of my site's pages using the /index.php/some-url structure, which is possibly why I was experiencing this issue more and more on my site... I'm quite not sure how this came to happen since my custom sitemap module doesn't contain any of these URLs.

I found that if you first clear all caches then visit an uncached page url like: /index.php/example-url on a Drupal site, all of the Drupal generated links on that page, like Menu links, will have URLs that also follow the same structure. And even when you visit the correct version of the page eg. /some-url you will find all/some of those corrupted menu/generated page links are still present. I don't know how/why but I'm guessing that these URLs are being cached somehow (although I couldn't find them in any cache bins), but maybe the server is doing that caching?

Also, if you use something like Metatag which adds a canonical link tag the URL that is generated for that will also follow the /index.php/url structure, which is obviously a pretty bad thing because then you have two different page URLs on your site which respond with exactly the same content.

Clearing the cache does fix the URLs, but this doesn't solve the problem from happening again.

Here are the fairly simple steps I used to "fix" it:

• Removal request to Google Search Console to remove all urls prefixed with /index.php/
• Update to robots.txt: Disallow: /index.php/
• Update to .htaccess to rewrite /index.php/ from any incoming requests:
  RewriteCond %{REQUEST_URI} ^/(index.php/)+(.*)
  RewriteRule ^(.*)$ /%2 [L,R=301,END]
• Flush All Drupal Caches

I'm no RewriteRule expert, and so no idea if this is a one fix for all, but this worked for me. Perhaps someone with more expertise can suggest a better way? Then maybe there's a consideration to make updates to these files in core. Unless again, I am terribly misinformed!

What do you guys think? Am I off the mark here?

I wanted to comment on our experience with this issue. We had a third-party application on our server that when users logged out, would send them back to our Drupal site but with /index.php appended to the URL. Because we had the redirect module set to "Enforce clean and canonical URLs" we never really noticed that the app was sending users to /index.php before redirecting to the base URL. But prior to getting redirected, Drupal kicked off the cron (we don't have crons configured at the server level, and instead we're just using the built-in poor man's cron). So the cron was getting triggered from /index.php/cron/[cron-key]. The issue was noticed in the sitemap.xml, which gets rebuilt on every cron run, where all of our URLs had index.php.

The solution is to prevent the site from being accessed at index.php at the server level.

#5 Works for me.

I just installed Redirect contributed module on the site.

By default this module has enabled "Enforce clean and canonical URLs." option.

Thanks @rick_p!!

#5 works for me with Drupal 9.3.0.

Thank you @rick_p!

This issue just popped up on our production site. I checked the caches as shown in #17 and found 425 entries that should not have been found.

Rebuilding the caches with "drush cr" fixed it.

Next week I'll explore the Apache config fix. (I am actually on vacation and was woken up to deal with the issue on our site.)

My configuration:

• Operating System: Windows 10 Enterprise and Windows Server 2012 R2 Standard (both are 64-bit)
• Web server: Apache 2.4.51 (64-bit) using mod_fcgid for PHP
• PHP version: PHP 8.0.13 (64-bit, non-threadsafe)
• Database: Microsoft SQL Server Enterprise 14.0.3048.4 (64-bit)
• Drupal version: 9.2.11
• Drupal driver for SQL Server and SQL Azure: 4.2.3
• Composer 2.24
• Drush 11.0.2
• PHP to SQL Server driver: 5.9.0 from https://pecl.php.net/ for sqlsrv and pdo_sqlsrv

I had the Problem starting from 9.0.0 on, but not constantly. I didn't find a logically reason.

the workaround #5 works for me now with Drupal 9.3.6 as well.

Thank you @rick_p!

This seems to be duplicating another issue.

Closed the following earlier issues as duplicates because this issue has the most discussion

• #2642126: [regression] Rewriting allows index.php/foo/bar URLs to function
• #2688632: index.php is inserted in menu links
• #2987492: NGINX - index.php append on images src / sitemap / css - Cache Rebuild Resolves

All the duplicates report the random nature of this problem. Clearing cache will 'fix' the problem but it will re-occur.

I am not sure the best component for this, so leaving at path.module.

I confirm we have same issues on Drupal 9.4.

• Only on production
• Random
• Cache clear temporarily resolves the issue, but after some time it returns.

Only on prod

i have this Probelm since 4 years with Drupal 8...

hasardous, but no make problem...

But, I migrate on Drupal 9 and index.php is put in link for my image link sometime, and message not found.... xD

I have Sitemap, remove, but no change, I have put the line on setting.php and works for the moment:

RewriteCond %{REQUEST_URI} ^/(index.php/)+(.*)
RewriteRule ^.*$ /%2 [R=301,END]

Add after RewriteEngine, not a the end of file.

i use Pathauto, but people here tell same problem with no module installed, soo, seem to be a problem in Core No? xD

@Amavi, thanks but "END" is not working because error:
"RewriteRule: unknown flag 'END'"

Instead, I use this:

RewriteCond %{REQUEST_URI} ^/(index.php/)+(.*)
RewriteRule ^.*$ /%2 [R=301,L]

Another report:

We have this too on a 9.3 production site, seems to happen on the same page but inconsistently. The links in question are part of a Views output, the problem only happens for anonymous users and not logged in users.

Neither sitemap (or xml sitemap) or redirect modules have been enabled

A cache rebuild temporarily fixes the problem

Enabling the redirect module with "Enforce clean and canonical URLs." fixes the problem

Try drush cc render

drush cc render worked the first time. It's appeared again... I want to say a week after doing the drush cc render. I believe the index.php has been around for a long while now, but it was only recently noticed to cause a problem with private area file downloads. A double index.php is being added to the url for those links. /index.php/index.php

Some days ago it happened to me too. 2 different websites.

Drupal Version
9.4.3
Server web
Apache
PHP 7.3.25
DB mysql 8.0.26

Drupal Version
9.3.6
Server web
nginx/1.18.0
PHP 7.4.30
MariaDB 10.5.15-MariaDB-0+deb11u1

No drush, no composer on both
I can't remember any operation I did on both websites at the same time, before what happened. Except that I used google search console to check both websites, which I seldom do in a short time range (2 days) but that can't be the reason.
Same situation as in #38. Menus aren't affected.

This is a huge problem for indexing the websites in Google, because there are hundreds of wrong urls and Google complains about it. This obviously affects the website's ranking.

Got alerted to this from Google Search console. #5 fixed it up for me - thanks all!

@Amavi and @thomaswalther , your solution saved my life today :-) It seems that the "Redirect" plugin is not available for D8 so it was not an option (this is not clear for me but Redirect branch 8.x apparently needs D9 or D10). I cannot believe that this issue has been ongoing for the last 4 years. Our site was working perfectly until recently when these index.php started to appear randomly ... I cannot point to a specific action that did start the problem, weird.
Pierre

@zedgre Yes, same for me... 2 different websites, same core, different web server, different modules... Randomly crazy.

I just experienced this too. Drupal 9.4.8 hosted on platform.sh, multisite. One site did have the redirect module installed and did not experience the issue. The other did not have refresh enabled and did. drush cc render did resolve it, but I don't know if it will recur. I've been on platform.sh for a few years now and it has only happened once.

Hello @amavi and @thomaswalther, sorry to reopen the post.
I have since a few days, in an Apache platform, the same problem of index.php in the URLs, which also branches to all the menu links.
For more information, I found "index.php" matches in only two db tables: 'local_location (63 records)' and "batch (1 record)".

The local version of the site does not have this behavior.

I would like to try your code to get rid of the problem, but in Drupal 9.4.8 I don't have "RewriteEngine" in settings.php as per your indication, while I have it in the .htaccess file (RewriteEngine on)
How can I do it...can I still copy it into settings.php?

Thank you very much if you can give me any pointers

To fix this, install the Redirect module, go to /admin/config/search/redirect/settings and activate "Enforce clean and canonical URLs".

---

The fix actually is really bad if you rely on multiple aliases for one entity. 😬

Nonetheless the bug suddenly disappeared after I enabled that option and then disabled it again, flush cache. 🧐

You are correct. I installed and enabled the Redirect module with the settings you reported and the index.php string disappeared.
Later, after doing the update to D 9.5.4 and then D 9.5.5 I first disabled the feature and then uninstalled the module, and now index.php no longer appears.

I am still left with, in the tables 'locales_location' and 'batch' the old references with the advice 'Version of Drupal where the location was found'.
I'm trying to figure out if I can empty these two tables without creating any operational problems.
Thank you very much for your advice

We solved this issue installing and configuring the HTTPS and WWW Redirect module https://www.drupal.org/project/httpswww
In addition to solving the index.php problem, it also redirects www and http which allows you to always have one session open and not several for registered users.

Hi @Jorgefernandes,
thanks for your suggestion. At the moment the problem seems to be gone after an update, but this module looks interesting for other functions as well.
Thanks again

It's still here! & #5 seems to work.

Drupal 9.5.9, PHP 8.0.28, DB 10.6.13-MariaD8-log
From D7 upgraded site, some 2000 nodes, lot of different modules.
Had different pictures in different tabs in same browser window.
There was pseudomenu made with View, in v.1 headings were w 'index.pxp' (and some messed up), in v.2 without.

#49 worked for me. Many thanks!

We're also affected by this issue but want to avoid additional modules as workaround. Therefore, we adjusted .htaccess to let apache perform the redirect.

I think we should raise the priority of this issue. Even if drupal is able to deliver /index.php/node/27, all requests to /index.php/sites/default/files/* return 404 Page not found!

Here's a patch that could be included in composer patches.

Added patch #57 to composer.json
and this helped me to solve the problem at downloading files from /sites/default/files folder (page not found error for urls like /index.php/sites/default/files/filex.pdf).

Hi :)

At my .htaacces the last Line is red with Visual Studio Code:

# Pass all requests not referring directly to files in the filesystem to
# index.php.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^ index.php [L]

So I put favicon Line in comment and Line with index.php (last line) become normaly, good Idea???

# Pass all requests not referring directly to files in the filesystem to
# index.php.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^ index.php [L]

I have had this issue with a couple of sites now. i.e. index.php showing on Plesk web-server with being served with both Apache and Nginx. As soon as redirect module is enabled the 'index.php' part of the url goes away.

Erratic.

#57 looks interesting.

(On plesk, favicon.ico is also used/loaded from admin ui when available now. see https://docs.plesk.com/release-notes/obsidian/change-log/#plesk-18057)

I have come up with an initial and unsophisticated solution to the problem. In the UrlGenerator class (core/lib/Drupal/Core/Routing/UrlGenerator.php).

On line 364 we see the following:
$base_url = $this->context->getBaseUrl();

When a user requests the homepage using 'www.yoursite.com/index.php' we are getting '/index.php' as the base URL. This is a special case where we are on the homepage and we can safely disregard this base URL. The unsophisticated solution is to simply empty this $base_url variable.

if ($base_url === '/index.php') {
      $base_url = '';
}

There may be a smarter way to handle this special case. For now, this code will prevent the bug from appearing.

I put this on my Robot.txt :

/index.php/*

And on the module "Path Auto" I check/Uncheck the option User and problem seem stop.... Strange...

Hello,

I have randomly the same problem.

Will there be a backport to Drupal 10 latest?

Regards

@ddavisboxleitner I don't think that solution would work if the Drupal site is not at the root of the host, for example, www.yoursite.com/SITE/PATH/index.php.

I just want to report that on D10 10.4.3
I have a had the issue with index.php appearing and solved that by updating the nginx config, to include a rewrite.

I could replicate the problem by clearing the cache and directly afterwards navigating to https://mydomainname/index.php
from then on all cached urls would include the index.php

But now I have a simular problem with
/core/install.php/ appearing,

I suppose I am going to include a rewrite for that one too.... I cant explain it

I think this has been mentioned already but I don't see this issue after installing the redirect module.

Hello,

I don't know how it can be done, but I've noticed that I've solved this problem by using an external cron job, launched by my server, rather than using Drupal's internal cron.

Regards.

The path module doesn't do anything with index.php so moving to base system.

Reiterating what #20, #26, #27, #29 ... etc have said: #5 seems to be working for us after about a month in production.

I have the advantage of this not being on a production system yet, so I can mess about with things a little more.

I transitioned our dev site from one server and URL to another. The new site has the index.php issue, the old one does not. I still have access to both unaffected and affected servers.

Installing the redirect module on the affected system solved the problem, but removing it caused the problem to return.

For me, the issue seems to only affect the admin_toolbar menu system. If I'm on /admin/content, all the content links seem fine, but the admin menu links (except for Back to Site) are prefixed by index.php.

Very interesting: I pulled the DB from the affected system to my local, and index.php's appeared in the links. But when I cleared the cache with drush cr on my local, the links returned to normal.

Clearing the cache on the affected site obviously does nothing.

Checking the Status reports of unaffected and affected site. Both running apache 2.4.62. No obvious differences in status page, no obvious differences in drush status page.

My thoughts after playing with this for an hour: If the issue can be "undone" by moving the DB to a new site and clearing the cache, it's suggestive that the site cache is somehow being polluted by some kind of local environment setting.

This is a long shot, but since the cache situation acts differently, could it be that some caches persist or are cleared, depending on environment? See #3540339: Rebuild all caches, including APCu, when clicking "Clear all caches".

UPDATE: No, I tried with the APCu cache clear method as well, and it made no difference.

More data. I tried installing this in a new AWS deploy with no database, and the index.php showed up immediately, without any database.

My previous statements about index.php showing up with the DB implies an issue that gets set up in the cache.

But this is definitely environment specific.

I might have access to one AWS ECS environment where this is happening always, and another AWS ECS where it is not, with the same codebase. Going to see if I can reproduce that this isn't working and gather data about the differences.

Installing the Redirect module doesn't work for me, because it sends my homepage into an infinite loop.

I want to solve this. I have an affected server, an unaffected server, and no care for modifying files in vendor and core to figure out what's the heck is going on.

I've looked at URL construction and dug all the way back to the root index.php and then back to vendor/symfony/http-request/Request.php.

The baseUrl returned from

$request = Request::createFromGlobals();

in the root index.php is polluted with /index.php on affected servers.

I tried digging down deep into createFromGlobals, and it looks like there's some weird shell game going on in vendor/symfony/http-request/Request.php where the baseUrl is being set.

Looking at the variables from the affected servers, it seems $server['REQUEST_URI'] is index.php on the affected server, and empty on an unaffected server.

I can (mostly) replicate the problem by adding:
$_SERVER['REQUEST_URI'] = '/index.php';
before
$request = Request::createFromGlobals();
(Though this does also seem to break the entire theming system)

My .htaccess files are the same on both servers.

This really feels like an apache not handing off the clearing of $_SERVER['REQUEST_URI'].

Looking at the answers above, I concur that some kind of REQUEST_URI issue. Solution #57 seems to work for me, I'm going to try it instead of the redirect module.

One last note. For some reason, a drush cr was insufficient to clear the index.php out of the cache, but clearing the cache using the UI worked. Would have thought that was a fluke but it's happened twice now.

+1 to #57. Seems to be working great.

Also running into this issue. Seems like #2753591: Fix mismatch of $base_url with Symfony request object could help solving this.

See my comment #13, in my experience happens when the very first request after a cache clear contains /index.php, then it will stick.

That explain it not happening in the ui but with drush cr, somehow your cache clear primes with /index.php, or the underlying system, or a cron or something...

I don't thank that Drupal caches the base-path, that's determined fresh from each request.

Based on what I've seen and read, I think what is happening is this:

1. cache clear - no pages cached
2. Frontpage is one of the first hits at /index.php - with some server configuration Drupal generates now wrong links which point to /index.php/
instead of /page
3. Users and bots are following wrong linkes to /index.php/
. Drupal wrongly generates the page with basePath /index.php
4. Wrongly generated caches end up in page cache.

I've added a test-case demonstrating the issue, as I described it in #79. (branch 3050261-test-case-2 in issue fork)

Disclosure: I used AI to help creating this.

I’m also running into this issue on a Drupal 10 site.

Bing has indexed a few URLs that include /index.php/ (for example: https://www.example.com/index.php/test-site). Everything works correctly in Google. This only happens for a few pages, not the entire site which makes me think Bing may have crawled those pages when /index.php was temporarily exposed somewhere (possibly through cached or sitemap URLs).

After enabling the “Enforce clean and canonical URLs” option in the Redirect module, those /index.php/... URLs now correctly redirect to the clean version (e.g., https://www.example.com/test-site). However, I’m not sure how or when Bing will remove the old /index.php URLs from its search results after these redirects are in place.

I had this same problem (D10.5.4) and manually applied the patch in #56. So far it seems to be working, I'll update this if there's a problem.

The tests are failing if unclean URLs are required. I don't think we can solve this problem without saying goodbye to unclean URLs in Drupal.

Do we have statistics how many Drupal 10/11 sites are running without clean URLs (URLs starting with index.php)?