Boost trying to crawl Unpublished nodes

Long story short if the node has a URL alias, Boost will try to crawl it if you have "Crawl All URL's in the url_alias table" selected. Looks like I should try to do some sort of join on the url_alias table and the node table to check if the node is published. I would want this to be yet another option since I've used the crawler in the past to hit millions of nodes and the extra overhead of the join would slow down an operation like that quite a bit.

Query needed

SELECT *
FROM url_alias
LEFT JOIN node ON Concat( 'node/', CAST( node.nid AS CHAR ) ) = url_alias.src
WHERE node.status = 1
OR node.status IS NULL

In case your wondering, boost will auto remove entries from the cache table if it detects a 403. This occurs when a node that was published goes unpublished.

Let me know if this does it for you

Using SUBSTRING might make it run faster... yes it does. This runs at an acceptable level 1.2M rows in less then a second. Previous query timed out after 300 seconds.

SELECT *
FROM url_alias
LEFT JOIN node ON node.nid = CAST( substring(url_alias.src, 6) AS UNSIGNED )
WHERE node.status = 1
OR node.status IS NULL

I can do this without the option :)

in case your wondering the original query was doing a join on node/555

The new query does a join on 555

This plus casting the url_alias as an integer made all the difference.

Crawl from sitemap.xml sounds like a good idea... put in a request for that.

What to crawl can sorta be set in a limited way with the boost configuration block. Setting push to FALSE is supposed to do it; but I think it would only work for things that are already in the cache. It was one of those "this might be a good idea" that never really took off from my point of view (I never use it).

The what to crawl feature would be an extension of the boost configuration block, allowing presets per things like node type and view name, as well as a custom way to handle expiration so nodes of this type will expire X etc... #622820: Expiration Grid - road map for this module

How large is your URL_alias table? I have a test db that has 1,284,788 entries in it; just curious if your running an even larger site.

committed

I just started using the crawler and ran into this issue.

Patch at #4 works nicely for me also for unpublished nodes, thanks.

On a related note, I have pathauto making aliases for user/[username] but the "access user profiles" permission off for anonymous users. So I get an "access denied" watchdog entry for each user/[uid] and each user/[uid]/track and each user/[uid]/track/feed.

so I need to do a query for

SELECT count(*) FROM permission WHERE rid = 1 AND perm LIKE '%access user profiles%'

If I get 1 back then crawl user/* if 0 then do not crawl user/*. Sound right?

Whatever you think is best. User pages aren't nodes yet so you can't run them through node_access, I assume.

I also noticed boost hitting my user pages. I do have path auto set up to make aliases for the user pages. I tried adding user/* inside the "Cache every page except the listed pages." textarea but no luck :(

I guess i could just shut off path auto for user names if there isn't any other way around this atm.

#4 is already in the latest dev.

patch is for user/* pages.
@TfR75 do you have something else controlling node access? are you using multiple languages?

committed above patch

patch to fix the above query... its been committed.

This is to be expected; I'm limiting what I keep track of in the boost relationship table, keeps mysql happier on large sites.

This has been committed; keep the bug reports up