Problem/Motivation

Currently there is no way to bypass the crawlers. For example if you have website which has facebook sso and according to the application we need to provide url which facebook crawls, enabling this shield module fails that crawling no matter if you add IP Address list or Domain

Proposed resolution

We can create textarea which will define crawler source and by pass it via adding one of the condition in middleware, so said crawler source will work as expected.

CommentFileSizeAuthor
#2 bypassing-crawl-implemented-3285819-2.patch2.93 KBabhaypai

Issue fork shield-3285819

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

abhaypai created an issue. See original summary.

abhaypai’s picture

abhaypai commented
Status: Active » Needs review
StatusFileSize
new bypassing-crawl-implemented-3285819-2.patch2.93 KB

Adding patch for this feature.

abhaypai’s picture

abhaypai commented
Issue summary: View changes
vbouchet’s picture

vbouchet
Location Toulouse
commented
Status: Needs review » Needs work

Hi @abhaypai,

It seems an interesting feature that you requested and provided a patch. Before including in the module, I think we should make it more generic and use the word "User agent" instead of crawler as it actually test the configured values against the user agent. We should probably display a warning message as the user agent is easily manipulable and so Shield can be by-passed using that method. Finally, we should add some tests.

japerry made their first commit to this issue’s fork.

japerry’s picture

japerry
Location KOMK
commented
Status: Needs work » Needs review

As mentioned in #4, the feature I think is worthy -- however the patch needed a bit of rework. I pushed a MR that creates the new feature and adds some tests along with it.

japerry opened merge request !37

  • japerry committed aa879bbd on 8.x-1.x 
    task #3285819: Flag "administer shield" as a restricted permission.

  • japerry committed b37f5a91 on 8.x-1.x 
    feat: Issue #3285819: Allow bypassing shield based on the request user...
japerry’s picture

japerry
Location KOMK
commented
Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.