After a node is deleted, the URL is not removed from the registry. Because of this, the URL is kept added in the purge queue causing continuous 404 errors in the log.

Proposed Solution:

While handling the request, the HTTP middleware, UrlRegistrar should delete the URL entries, which are not returning response code 200.

CommentFileSizeAuthor
#6 urls-are-lingering-2875003.patch4.04 KBnielsvm
#2 urls-are-lingering-2875003-2.patch1.83 KBparyank

Comments

paryank created an issue. See original summary.

paryank’s picture

paryank commented
StatusFileSize
new urls-are-lingering-2875003-2.patch1.83 KB

Uploading a patch with the fix.

paryank’s picture

paryank commented
Status: Active » Needs review
nielsvm’s picture

nielsvm commented

I'm looking into this, thank you!

nielsvm’s picture

nielsvm commented

I just tested this patch on a clean installation with an empty registry... and sadly, it is preventing items from ever getting registered in the first place, and would therefore break the entire module.

The reason is likely that it takes all FALSE return values from ::decide() as reason to remove URLs, including when URL hits are getting denied which are inevitably preceded by a URL miss which must get cached. This means this will need a rework and I will spend some time on that today as I do like the general idea here.

nielsvm’s picture

nielsvm commented
StatusFileSize
new urls-are-lingering-2875003.patch4.04 KB

New patch, changes:

  • ::decide() renamed to ::determine()
  • ::decide() now returns FALSE for responses that should be ignored, and NULL for responses that must be removed.
  • ::remove() now shielded against empty calls.
nielsvm’s picture

nielsvm commented

Did some serious testing and it now seems to work as intended. After I had spidered 509 URLs into the registry, I mass deleted all the nodes from Drupal's content pane. After that, spidering only touched pages of the side that now remained and therefore didn't affect the registry, it stayed at 509 URLs.

However, when I took all the URLs from the registry and scripted curl commands against them, the registry quickly shrunk to well below 50 URLs. In other words, we can't make this perfect but it now seriously helps maintaining the registry a bit more actively.

Committing, thanks so much for this idea and patch!

nielsvm’s picture

nielsvm commented
Status: Needs review » Fixed

  • nielsvm committed 815ca64 on 8.x-1.x 
    Issue #2875003 by paryank, nielsvm: URLs are lingering in registry even...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.