Problem/Motivation

Coming here from contrib: #3494959: Add the nofollow attribute to the rel attribute of the "Show More" link I came to the conclusion that instead of adding a setting in each pager it would make sense to add a boolean setting to set rel="nofollow" attribute on views pager links.

Drupal’s Views pager plugins should provide a general option to add rel="nofollow" to pager links. This is not only helpful for "load more" pagers, but also for regular pagers in many cases.

Views often list up contents that should get indexed, while the listing view can have a lot of pages and search engines often should only index the first page of a listing, as it represents the canonical, authoritative version of the content collection.
However, deeper paginated URLs frequently offer little unique value, can dilute ranking signals, and may waste crawl budget. A nofollow attribute on pager links allows search engines to index the main listing page while discouraging crawling of subsequent pages.

Global meta directives (like noindex, nofollow) cannot solve this problem when Views are embedded inside other content, because meta tags apply to the entire page, not to a specific View. A pager‑level rel="nofollow" setting would work consistently whether the View is a standalone page, a block, or embedded (eventually with other views) in a node.

Adding a simple checkbox such as “Add rel="nofollow" to pager links” would be fully backward‑compatible, easy to implement, and aligned with Drupal’s philosophy of giving site builders granular control. It would reduce the need for custom preprocess functions or custom template overrides.

Steps to reproduce

Have a view with many items and pages. While the page itself should get indexed, e.g. "Archive", the subsequent pages should not get indexed and especially should not get crawled. This is even more important for infinite scrolling pagers (but also for all others)

Proposed resolution

Add a boolean pager “Add rel="nofollow" to pager links” setting which adds the rel="nofollow" attribute on pager links

Remaining tasks

  1. Discuss
  2. Implement
  3. Test
  4. Release

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Comments

anybody created an issue. See original summary.

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

If maintainers agree, we'd like to implement this, because we think it would be very valuable.

scott_euser’s picture

scott_euser commented

There is risk in doing that I think; from https://developers.google.com/search/docs/crawling-indexing/links-crawlable

Every page you care about should have a link from at least one other page on your site.

I.e, you cannot rely on sitemap.xml alone, so there is risk that a content item loses its only internal link if pagination has nofollow on it.

Probably its better in contrib space to add nofollow when e.g. exposed filters or other query string vars exist so that pagination without filters leads to the internal link requirement being satisfied without exponentially increasing the number of indexed pagination pages when filters are applied (in case those are discovered as unique URLs)

So I think this is probably a 'wont fix' as currently considered

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Good point @scott_euser, thank you! You hit the spot.

Do you think it should rather be a "nofollow" on the pager links to also reduce (BOT / AI) traffic or a noindex header on these pages with filtering present?
I guess the hard thing in contrib will be to safely identify these cases.

scott_euser’s picture

scott_euser commented

Maybe both actually.... Probably safest in contrib is to make it a per view setting that can be enabled; might need to go in the advanced tab rather than pager as it's easier in an preprocess hook to generically apply. Ill see if I can contribute this as we did it for one site where less advanced crawlers got stuck in an endless/seemingly infinite crawl because of the large number of possible filters

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Thank you very very much @scott_euser! I'd of course be willing to join with the DROWL team, we ran into similar trouble.
I couldn't find a fitting solution yet and I'm unsure how to start and solve it best.

Feel free to ping me here or via contact form or @grevil via Slack (I'm not active on slack). Once we have the contrib module present, let's close this?

scott_euser’s picture

scott_euser commented

Sounds good; I'm thinking a module name like "Views Crawl Control" making module clear its aim is to help well-behaved bots crawl and index optimally but that other modules like Protect Views Flood Control, other flood control modules, and non-Drupal solutions like Cloudflare rate limiting should still be in place for malicious intents to e.g. avoid cache and overwhelm a server.

I think the scope for the module would be to handle a few things:

  1. Ensure nofollow on pager links that are more than just ?page=*
  2. Ensure canonical to no query string if more than just ?page=* (but keep e.g. ?page=2, or ?page=3 as canonicals)
  3. Ensure noindex tag similarly for beyond just ?page=*
  4. Support multiple pagers on same page (so dynamically getting ?page per view)

I got Fable to validate/create a plan while we still have the bonus fable credit, but wanted to check if you roughly agree (mostly on naming since the rest is easily changeable). I can add you and @grevil as co-maintainers if you'd like

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Thanks @scott_euser I like your plan! The name also fits, I think. I probably would have chosen something around "Robots", but that might be too specific. Yes, feel free to add us and let us know if we can help.

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

One thing just came to my mind, which might really make sense: Make the base module less specific and the views integration the first submodule!

So the name might be "Crawl control" or "Robots control". And the submodule might get a : Views suffix.

I had to remember a case different from views, which is still highly related: Setting noindex on URLs with certain URL parameters by whitelist or blacklist. I currently don't clearly remember what the exact use-case was and I'm unsure if it should not better get solved through "canonical" meta tag. Still I think it makes sense to give us the option to provide broader functionality around this?

scott_euser’s picture

scott_euser commented

Sure; could start with Crawl Control, but maybe avoid sub-module at first (since that isn't always obvious when people install); we can always move later if the scope of non-Views grows large (update hook can enable sub-module to make it seamless)

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

100%

scott_euser’s picture

scott_euser commented
Status: Active » Closed (won't fix)

Okay I had a first pass at it here https://www.drupal.org/project/crawl_control - I think we can continue there in issues and take it forward. Will close this one off. Thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

WHAO, impressive work @scott_euser! Just had a short look and you even already test the Metatag combination case - which would have been my next check.
README is also fantastic!

The only thing I'd vote for is maybe adding a sentence at the very beginning of the module page that currently it's focused on views?

Do you see any downsides or risk in using this module in general? If yes, we should list them in the README and module page. I couldn't find one yet. So for projects using filters on public views using this module ALWAYS makes sense to me?

Looking at the issue summary here: What about the (Core) "Show more" link? (#3494959: Add the nofollow attribute to the rel attribute of the "Show More" link)

Anything else we should look into? Should we add a feature request for facets support? Eventually as submodule then?

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

PS: I think it would be VERY helpful to have a page to set the global defaults for new views and bulk-apply them to selected existing views?

Maybe that page could also be linked in the .info.yml as configure and inform users where to override the settings in each view?

Writing this I think for better UX it might be best to have something like "[ ] Use global defaults" or "[ ] Override global defaults" plus the option to simply disable the functionality for a view in the view display extender and define the defaults centrally?

Sorry for writing down my ideas as they come to my mind... at a brainstorming state.

scott_euser’s picture

scott_euser commented

Yeah that sounds good - global opt in (or opt out) with per view override sounds sensible

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Do you see any downsides or risk in using this module in general? If yes, we should list them in the README and module page. I couldn't find one yet. So for projects using filters on public views using this module ALWAYS makes sense to me?

Maybe it would make sense to give a short answer to "Who is this useful for?" I think that might make it easily understandable.

And I think the use-case is public views with multiple / many result pages and filtering / sorting enabled?

Afterwards the problem solved could be explained?

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

P.S.: I don't want to interfere while the work is in progress, so please excuse me for writing all this here instead of doing it myself. You should be the one to make the final decision on this.

scott_euser’s picture

scott_euser commented

All good, happy for you to raise ideas and feel free to edit things like texts and screenshots, etc; I'll focus on the code for now