Problem/Motivation

It'd be nice if the user could specify the base URL to use for the index.

This is useful if you're running a headless CMS running under a different domain than the public facing one, and would like changes to the entity to trigger an index under a specific domain.

Steps to reproduce

N/A

Proposed resolution

1. Provide a configuration for setting the base URL to use for the index link.
2. Add a special index_now option to the URL so that specific outbound processors (built into the module, or custom) may switch the base URL as necessary.

Remaining tasks

Provide MR.

User interface changes

There might be a new configuration for specifying the optional base URL to use.

API changes

N/A

Issue fork index_now-3586683

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

codebymikey created an issue. See original summary.

macsim’s picture

Hi codebymikey

Thank you for opening this issue and for taking the time to provide a merge request - the use case is clear and the overall approach is well thought out.

The idea of using an OutboundPathProcessorInterface tagged service is idiomatic Drupal and a clean way to handle this. Marking URLs with an index_now option to scope the processing is also a good pattern that leaves room for custom processors on the site level.

A few things to address before we can move forward:

  1. Unused service argument
    @index_now.apikey.manager is declared as an argument in index_now.services.yml but IndexNowOutboundPathProcessor has no constructor and never uses it. Either the argument should be removed from the service definition, or there is a missing constructor - could you clarify the intent?

  2. Alias URLs won't be processed
    The condition in processOutbound includes !empty($options['route']), which excludes unrouted URLs. However, PathAliasOperations::buildAliasUrl() uses Url::fromUri('base:...'), which goes through UnroutedUrlAssembler - $options['route'] will never be set for those URLs. The index_now option is correctly set on them in the MR, but the processor will silently skip them.

    Since UnroutedUrlAssembler does honour $options['base_url'],
    removing the !empty($options['route']) guard should make the override work for both entity and alias URLs. If there is a reason to keep this distinction, it would be good to document it.

  3. Minor: no processor priority set No processor priority needed
    The path_processor_outbound tag has no priority attribute. This is not blocking, but worth being explicit about in case other processors on the site also manipulate base_url.
    The path_processor_outbound tag intentionally has no priority set. This processor only acts on URLs explicitly flagged with index_now, so ordering relative to other processors is not a concern - no other processor will compete over base_url for these URLs.

  4. Fix PHPCS errors and warnings

Once these points are addressed we can move forward with a proper review.
Thanks again for the contribution!

codebymikey’s picture

Good points! And thanks for addressing some of them for me. I ran into the alias issue as well, and was in the process of creating test coverage for the alias override code, but didn't get round to finishing it off yesterday.

I'll try rebase against your version, then add the remaining test coverage sometime today.

Edit: just saw your final push, I guess that's everything (I don't the think a priority on the path processor is strictly necessary as the default is 0 anyway, and those needing to override it can add a higher priority if they need to)

macsim’s picture

Status: Active » Needs review
macsim’s picture

Everything seems to be settled and covered now.
If you have a testing environment, feel free to use it :)

I'll make a second MR for the 4.0.x branch.

macsim’s picture

The automated test suite covers the path processor for routed URLs (entity URLs)
and buildAliasUrl() in isolation. One scenario is not covered by
automated tests: the full end-to-end pipeline from an entity save to the actual
IndexNow API request.

Before merging, it is recommended to manually verify the following:

  1. Add the following to your settings.local.php:
    $settings['index_now.base_url'] = 'https://other-domain.com';
  2. Enable the IndexNow logging (Verbose mode).
  3. Create or update a node.
  4. Confirm that the URL sent to the IndexNow API uses
    https://other-domain.com as the base, not the internal Drupal
    domain.
  5. Create or update a path alias for that node and confirm the same.
macsim’s picture

Status: Needs review » Needs work

Got an error on the 4.0.x branch while manual testing (and I am pretty sure the same behavior would occur on 3.1.x):

Index Now returned an error: Client error: `GET https://www.bing.com/indexnow?url=https://other-domain.com/node/1&key=40...`
resulted in a `422 Unprocessable Entity` response:
{"errorCode":"InvalidRequestParameters","message":"Keylocation is not allowed.
Please ensure the keylocation url belongs (truncated...)

The keyLocation domain is not the same as the indexed URL's domain.

#3557839: Allow Users to alter URLs submitted to Index Now and Key Location URL could do the trick but it would need a developer to write code.
Alternatively, we can use $settings['index_now.base_url'] to also build the keyLocation URL when the setting is configured.

However, Drupal won't be able to serve the API key file on the other domain - it will need to be created manually there. We will need to document this clearly in README.md.

macsim’s picture

Status: Needs work » Needs review
macsim’s picture

Status: Needs review » Reviewed & tested by the community
StatusFileSize
new77.24 KB

Manual testing is ok now
Going to merge on both branches.

Thanks again codebymikey

  • macsim committed 3308d2ca on 3.1.x authored by codebymikey
    feat: #3586683 Provide a way to set a custom base URL for the index
    

  • macsim committed d71091fb on 4.0.x
    Issue #3586683 by codebymikey, macsim: Provide a way to set a custom...
macsim’s picture

Status: Reviewed & tested by the community » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

codebymikey’s picture

Thanks for the swift feedback and subsequent implementation Maxime!

I've applied the patch onto the site and will open a follow-up issue if there are any more issues.

macsim’s picture

You're welcome.
I also reopened a postponed issue and implemented a hook to handle advanced use cases where URL patterns differ between the Drupal backend and the frontend domain: #3557839: Allow Users to alter URLs submitted to Index Now and Key Location URL

For example, if your Drupal instance serves content at /node/123 or /en/article/my-slug but your frontend (decoupled, reverse-proxy, CDN, etc.) exposes those pages under a completely different URL structure, a developer can now implement hook_index_now_urls_alter() to rewrite the submitted URLs accordingly - ensuring that what gets sent to IndexNow always reflects the actual public-facing URLs indexed by search engines.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.