When the website is using multilanguage with language prefix the hook_menu 'robots.txt' gets redirected to "domain/[prefix]/robots.txt".

Issue based on: https://www.drupal.org/project/robotstxt/issues/2930799

CommentFileSizeAuthor
#9 interdiff-3026573-8-9.txt321 bytesalex.bukach
#9 robotstxt-avoid_redirection-3026573-9.patch520 bytesalex.bukach
#8 interdiff-3026573-1-8.txt454 bytesalex.bukach
#8 robotstxt-avoid_redirection-3026573-8.patch522 bytesalex.bukach
#4 robotstxt-avoid_redirection-3026573-1.patch531 bytestbsiqueira
#3 robotstxt-avoid_redirection-3026573.patch664 bytestbsiqueira

Comments

tbsiqueira created an issue. See original summary.

tbsiqueira’s picture

tbsiqueira
He/him
commented
Issue summary: View changes
tbsiqueira’s picture

tbsiqueira
He/him
commented
StatusFileSize
new robotstxt-avoid_redirection-3026573.patch664 bytes

Proposing patch to fix the issue, the fix is based on the already proposed patch at the related issue on this bug description.

Implement robotstxt_url_outbound_alter and only alter prefix if we are under /robots.txt path

tbsiqueira’s picture

tbsiqueira
He/him
commented
Status: Active » Needs review
StatusFileSize
new robotstxt-avoid_redirection-3026573-1.patch531 bytes

Proposing an even simples solution.

Anonymous’s picture

Anonymous (not verified) commented

#4 works for me.

Another issue:
When xmlsitemap is used only the system sitemap in the original/system language is added in the last line of robots.txt.

Eg, I have a multisite EN (1st language) and NL, "Sitemap: https://www.domain/en/sitemap.xml" is added.
I am missing my second language sitemap: "Sitemap: https://www.domain/nl/sitemap.xml"

Is it possible to add all the used languages as sitemap link?

Anonymous’s picture

Anonymous (not verified) commented

Maybe the robots.txt url's with language prefix should not exist (or hidden)?
Only the robots.txt without language prefix is used.

ccjjmartin’s picture

ccjjmartin commented
Category: Bug report » Feature request
Priority: Major » Normal
alex.bukach’s picture

alex.bukach commented
StatusFileSize
new robotstxt-avoid_redirection-3026573-8.patch522 bytes
new interdiff-3026573-1-8.txt454 bytes

Thanks for the patch @tbsiqueira! Generally it works for me, however when I am adding more links via hook_robotstxt() and call url(), it strips the language prefixes from those URLs too. Here's a corrected patch.

alex.bukach’s picture

alex.bukach commented
StatusFileSize
new robotstxt-avoid_redirection-3026573-9.patch520 bytes
new interdiff-3026573-8-9.txt321 bytes

Actually, there's one more issue: some modules (like globalredirect) expect $options to have the prefix key since it's initialized by url(). Therefore let's better reset it to the default value instead of unsetting.

ccjjmartin’s picture

ccjjmartin commented

I know this is 7.x and not 8.x but I am committing the solution in this thread to the 8.x branch: https://www.drupal.org/project/robotstxt/issues/3157451 (adding some information)

keboca’s picture

keboca commented

@ccjjmartin thank you, I was looking for the solution to 8.x branch