I've been digging into a goofy problem for weeks, no solution so posting in case someone can shed light. I'm reporting it here because at least part of it is explicitly reported as related to Domain Access.

Domain Access seems to be working fine in all aspects I can see. It is handling a bunch of subdomains. (I used it with D5, and recently added it in D6, so I think I know how to manage it.) But...

In many cases, when Googlebot is the site visitor, my Drupal log often gets 2 sequential errors:

Domain access failed to load during phase: -1
I haven't determined the cause of this error message. DA is configured just right from all I can tell, and always has been. Everything about it seems to work fine. If I go to the URL manually there's no error, I see the correct domain, etc.

Followed by 3 or 4 instances of:
You have an error in your SQL syntax...
I can see that this error is truly a SQL syntax problem. In the compound SELECT statement, one clause has "gid =" nothing, no value, which is not valid SQL. Sometimes it shows as "na.gid =". However, I can't determine where this "bad" SQL arises. Manual exploratory visits to the problem URL and related URLs don't trigger any errors.

When I go to the exact links reported by Drupal as triggering the error, the page displays fine and I get no errors. I can do this with Firefox or IE, whether authenticated or anonymous. So, I have no clue why Googlebot triggers errors that I can't recreate, and I don't see them arising other visitors either.

Just in case, I resaved modules. I re-ran Update. I checked and resaved permissions. I rebuilt all node permissions. I cleared all caches. No errors reported from these actions.

I have no code that would intentionally/knowingly mess with a Googlebot visit. I assume none of the DA modules have browser/user-agent conditional behavior.

So, it's a mystery. Anyone have a suggestion? Should I post detailed examples of the error reports?

Also -- don't know if it is related -- I sometimes get similar errors when MSNbot visits, though it really messes up the log because it frequently tries to go to URLs that I haven't head in years -- a decade in some cases (My main domain has been online since 1994). The log gets so messy that it's difficult to discover current problems vs. the fact that it seems to ignore 301 responses year after year, perpetually revisiting long-gone pages.

Running
Domain Access 6.x-2.4, also same version of Domain Alias, Domain Configuration, Domain Content, Domain Navigation, Domain Settings, Domain Source, Domain Theme, Domain Views. (I'll probably disable Domain Theme, not needed. For a while I had Domain Prefix active but never used it so uninstalled. Domain Strict and Domain User are also in modules list but never activated.)
Drupal 6.16
MySQL 5.0.87
PHP 5.2.11
Apache 2.2.14

Files: 

Comments

hawkdrupal’s picture

More...

Sometimes, the URL Googlebot visits that triggers the MySQL error due to a missing gid value really does have a problem. But the problem is equally mysterious.

When the node is visited by an anonymous user, the result is Access Denied. Yet there is no explicit access control on such nodes. If I visit the page as Admin and simply RESAVE the page, the problem clears up. I see nothing on the page that is set wrong, I change nothing, but saving it again fixes the problem that tripped up Googlebot and other Anonymous users.

This happens no matter which subdomain is used to visit, including my default domain.

Note that all such nodes are set to be included in all "affiliated" domains, and they were never knowingly set otherwise. Also, there were no evident Anonymous access problems before DA was installed.

I don't use (and haven't used) Domain Access to control access, just to let me control each domain's home page and subdomain navigation. Yet something is likely checking for domain access control and thereby failing. (Perhaps this is source of the missing "gid" value?)

Almost all of the problem nodes were previously edited using a batch mode to set a few things -- but not/never anything seemingly related to access control.

I'm still discovering the problem nodes, and resaving as I find them via the error log. I'd like to fix this globally, maybe via manual SQL against all the nodes that might be affected, but need guidance.

Maybe this situation is the BIG clue to why Googlebot triggers errors?

agentrickard’s picture

The first error message (phase -1) means that DA failed to load during settings.php and the $_domain could not be determined.

Typically this happens when a site pings you by IP instead of HTTP_HOST. Googlebot may be doing that. You can enter a domain alias for the IP address that should clean up that problem.

The second error should correct itself when the first is cleaned up.

hawkdrupal’s picture

SOLVED:

Module GlobalRedirect was set to strip off trailing /0 to clean up the way Drupal sometimes puts this on the end of a URL yet with/without is the same page. However, it breaks Domain alias editing. Since this seems to be a popular module, a suggestion is to change the domain editing URL's final path segment to be a word plus the domain, such as /admin/build/domain/alias/edit-0, .... /edit-1, ... etc .

Below is the problem with Domain Access that stopped me cold until I solved it:

------------------

Thanks for the suggestion. But to try it, I ran into a new problem -- and either I'm crazy (probably) or I'm hoping you can help me get past it.

I open Domains (/admin/build/domain/alias) to add my server's IP as an alias. The alias I want to add (per what Googlebot seems to be doing) is to my primary/default domain 0. But I can't.

If I click "edit alias", which is path /admin/build/domain/alias/0, the screen flashes but I end up still sitting on the Domains list page. I'm clicking "edit alias" because this domain already has one alias, for the domain name without www, but the same problem happens if I click "aliases" in the action column.

If I type the same URL into the browser, not relying on clicking a link, same problem. I type in .../alias/0, the screen flashes, and I'm back at .../alias/ ... no zero.

It seems something is redirecting or chopping off the /0 part. I can click and add/edit the aliases of any other domain, just not domain 0.

Just in case, I checked my system for inadvertent Apache redirects and Drupal path aliases but find nothing that could do this. Also, I tried to go to /admin/build/domain/alias/0 from one of my subdomains in case it's not allowed to change the current domain, but no luck. And Batch updating doesn't seem to do domain aliases.

So I don't find a way to implement your suggestion.

Is domain 0 a special case, somehow protected? Or could there be a setting somewhere I should change? Or must I go straight into the database and hack it to add an alias for domain 0?

hawkdrupal’s picture

BASIC PROBLEM UNSOLVED:

Alas, spiders -- specifically msnbot the past couple of hours -- are still triggering the SQL error due to na.gid = no value, even with the server's IP set as an alias.

Yet the same URL that has the "error" works fine when visited in a browser.

agentrickard’s picture

I don't like Global Redirect for this very reason. Since it seems to be chopping the 0 without much thought. You might file the bug over there. No reason for that module to be chopping a 0 off an admin URL, since search engines don't spider /admin.

hawkdrupal’s picture

I was able to "fix" Global Redirect by turning off the chop-off-0 mode, but you're right, it's brain dead.

Back to my real problem, I added my IP as an alias as you suggested. I assume this means just put in the IP, without a protocol prefix.

But I'm still getting the two errors from Googlebot and MSNbot. The key fact is that the same URLs do not generate errors when visited manually.

Domain access failed to load during phase: -1. Please check your settings.php file and site configuration.
In my logs I don't find this error triggered by anything but a crawler.

Followed by 3 or 4 instances of:
You have an error in your SQL syntax...
This is usually due to "gid =" nothing, no value, or "na.gid=" nothing.

Not much of a clue, but *sometimes* this string of errors arises when an Anonymous (or even non-Admin) user does not have access to the targeted page, or the page doesn't exist (MSN loves to go to ancient URLs), so the crawler instead gets a "access denied" or "page not found". But... going to the same URL manually doesn't trigger the Domain Access errors.

Is there something else a bot does that a browser does not do that is not compatible with Domain Access? For instance, is Domain Access expecting some "standard" user-agent info that a crawler might not provide?

agentrickard’s picture

hawkdrupal’s picture

Just want to let you know that the problem persists.

Googlebot and MSNbot trigger the above Domain Access errors constantly, but the same "bad" URLs visited manually do not trigger the errors.

In fact, we've never found a human visitor, or even another crawler, generate these errors.

Since it's a string of errors, would it be useful to have Domain Access issue a "go away" code on the first error (Domain access failed to load during phase: -1...) -- so the bot doesn't keep hitting the link, just generating a series of SQL errors? (Maybe this already happens...?)

agentrickard’s picture

But how can we identify a bot vs. a real visitor?

Why don't you just ban those IPs?

The error message is for you, the server admin, so you can fix the installation. I still bet this is an alias issue, with the crawlers using IPs or an unregistered domain to hit the site.

hawkdrupal’s picture

But...

I don't really want to block major crawlers, just get rid of the errors. Normally it's straightforward to identify crawlers via the User-Agent value -- essentially the bot's name -- that they pass to the visited target. Google, Microsoft, Yahoo, and other major bots seem to do this reliably.

The log shows the URL the bot is hitting that generates the Domain Access error, and it's a valid URL when visited manually. Of course, if Apache and Drupal are not accurately recording the visited URL in the log, I don't have anything to go on...

Feature Request:

While I have no evidence that crawlers are hitting the server via mystery IPs or domains, I can't disprove this. I started this site in 1994 so who knows what "bad" links to it might exist out there. But after many, many, many hours of digging I don't find anything to "fix", especially since the logs don't indicate that the bots are hitting "bad" domains. Only Domain Access is reacting negatively.

Idea: Would it be feasible for a Domain Access to have an option to handle this gracefully -- by using a default domain/alias (domain 0 ?) rather than issuing a raw error which then leads to a cascade of SQL errors? I imagine this could be especially helpful to admins when a domain gets retired/changed.

This would be similar to how most e-mail systems allow a catch-all address to receive mail sent to non-existent/defunct addresses.

And in a perfect scenario, Domain Access responds to "no such alias" by instead using the specified/default domain, AND responds with a 301.

agentrickard’s picture

Well, we _do_ that when the bootstrap file is loaded normally from settings.php. See domain_init() after the error handling.

You're having a separate problem in that settings.php doesn't seem to be loading at all. Perhaps the bots are hitting 404 pages that are themselves calling page elements that require a partial bootstrap.

You could try adding a line to settings.php that provides HTTP_HOST if that value is not present, for some reason, but Drupal should already have handled this.

You doing anything special at the server level?

We could set domain_id to 0 in the event of an error. Then the warning would persist, but not the SQL errors.

hawkdrupal’s picture

Well, I dislike mysteries, and it seems you do too. But that's what I have.

Nothing odd in settings.php (which is essentialy unmodified except to add your .inc line) -- evidenced by how everything works for normal users. We only get Domain Access error except for *some* visits by Googlebot and mostly by MSNbot (which seems to hit us almost constantly). And likely we don't get these errors at every bot visit, just some, yet the reason is not evident.

Nothing odd with our Drupal/Apache server that I know of, unless a bot requires the server to respond differently than a browser does, therefore requires something special set up in Apache.

Settings.php is in sites/mydomain.com, which Drupal finds by itself. All the other domains controlled by Domain Access are subdomains. But almost all of the DA errors are when the bot is visiting our original/domain-0, which is www.mydomain.com. (Again, the "error" URL reported in the log is valid and works manually.)

We are not using DA's domain-level access control (so that module is not activated), so actually the URL's that trigger the error work at www.mydomain.com AND at any of the other subdomain.mydomain.com that are set up in Domain Access.

Re 404, Drupal's standard htaccess redirects this to index.php, which in-turn passes it to a specified 404 error page. That seems to work fine if I intentionally go to a non-existent page. (But difficult to probe its behavior in this situation because the DA "error" pages really do exist.) I wonder if Drupal's internal mechanism to invoke a custom error page is somehow goofing up -- but that would be only with these bots.

Note that Drupal's custom error page config (/admin/settings/error-reporting) shows DA's "Domain-specific settings" list. It defaults to www.mydomain.com, my domain 0. I hope I don't need to resave this page for each of my subdomains. I don't think so because many of the bot errors are with www.mydomain.com.

Hard to test a negative, but I'll try: I just removed the setting to invoke a custom error page, in case that's a factor. So now I see Drupal's default error page, which is a page with partially my theme (top and bottom but not side sections) plus the statement "Page not found". So I'm still trusting that Drupal is properly reporting 404. Given how often MSNbot visits (last DA error was 10 minutes ago), I'll probably know with a few hours whether the DA-related 404 errors continue to appear in the log, thereby removing the custom error page from the suspect list (but not exonerating Drupal's error handling).

What I don't know is in which order the error/404 steps happen -- when does Drupal report 404 to the visitor vs. continue to step through its logic until it finally displays an error page, and at what point in this flow does Domain Access enter the process. Maybe a bot reacts to a 404 differently than a browser (perhaps lacks the full two-way dialog process), which would explain (sort of) why the "error" bot URLs reliably work in a browser. Maybe Drupal's 404 process, interacting with a bot's behavior, goofs up the path so settings.php can't be found -- but only for the bot.

agentrickard’s picture

Category:support» bug
Status:Active» Needs review
StatusFileSize
new3.89 KB

Odd. At any rate, I'm pretty sure the SQL error is caused by changes to Views and Drupal core, and I think we can account for it. What I'm thinking is silently set domain_id to 0 on error, and give you the option to manually disable the warning message using $conf['domain_hide_errors'] = TRUE in settings.php.

I think that should solve your problem and makes sense as a safeguard.

Here's a patch to test. Makes sense to me.

hawkdrupal’s picture

Category:bug» support
Status:Needs review» Active

I really appreciate the quick response and patch.

However, I can't seem to apply it. The patch seems to be looking for newer versions of the two files, but most important, the versions I have don't seem to have the exact code that must be patched. I believe I'm using the files provided with Domain Access 6.x-2.4. Is this not the current 6.x version?

domain.bootstrap.inc -- looking for 25 Apr 2010 but I have 13 Feb 2010.

domain.module -- looking for 27 Apr 2010 but I have 18 Mar 2010.

I searched the files for the code section to patch manually, but didn't find enough of a match to be sure what to change.

So, can you either provide a patch for the versions I have, or if appropriate provide (or tell me where to get) the versions you are patching, or even better provide the patched files?

hawkdrupal’s picture

I disengaged Drupal's custom error page option a few hours ago, and just got another visit from MSNbot which generated the same error: Domain Access, "Domain access failed to load during phase: -1" but the URL cited in the error record is already my domain-0, and the URL works fine in a browser.

agentrickard’s picture

Category:support» bug
Status:Active» Needs review

The patch is against 6--2 dev. You'd have to do a line-by-line comparison against the version you have installed. Mainly in domain_init() and in moving the two functions from domain.bootstrap.inc to domain.module.

This is still a 'needs review'. Your inability to apply the patch doesn't change that.

agentrickard’s picture

Status:Needs review» Reviewed & tested by the community
StatusFileSize
new5.3 KB

Committed to 6--2. Needs to be applied to 7.x.

agentrickard’s picture

Status:Reviewed & tested by the community» Fixed

Committed to HEAD.

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.