Apache Solr Search Integration environment deleted when rpc.acquia.com is inaccessible [#1778282]

Although related to #1778072: Acquia Search credentials are lost when the site fails to connect to rpc.acquia.com, the root cause is slightly different and it is more critical in nature since the environment and environment settings are completely wiped out whenever rpc.acquia.com cannot be reached. Marking as critical because not only does this break search until the next cron run, but if your index set to "read only", it will come back up with read/write access since that is the default setting. Obviously this can potentially cause index corruption.

In order to replicate this bug, take the following steps:

Connect to Acquia Search using the standard install process, verify that it works
Change the "acquia_network_address" variable to something bogus, i.e. "https://example.com".
Run cron. Your environment and credentials are now deleted.

There is a corresponding bug posted against Apache Solr Search Integration that, under certain circumstances, will whitescreen your site when this condition occurs. See #1778150: Fatal error on all admin/config/* pages caused by SQL error in apachesolr_environment_load_subrecords() for more info.

Comment	File	Size	Author
#8	acquia_search-environment-delete-1778282-d6-8.patch	507 bytes	coltrane
#8
#3	acquia_search-environment-delete-1778282-3.patch	1.32 KB	coltrane
#3
#2	environment-delete-fix-1778282-2.patch	677 bytes	cpliakas
#2
#1	environment-delete-fix-1778282-1.patch	669 bytes	cpliakas
#1

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

cpliakas CreditAttribution: cpliakas commented 7 September 2012 at 18:57

Status:

Active

» Needs review

File	Size
environment-delete-fix-1778282-1.patch	669 bytes

The attached patch is a hotfix that only deletes your environment when the subscription is not found or expired. I'm not sure it is the best long term solution, but it more or less stops the bleeding immediately.

Comment #2

cpliakas CreditAttribution: cpliakas commented 7 September 2012 at 18:58

File	Size
environment-delete-fix-1778282-2.patch	677 bytes

Same patch with a better comment.

Comment #3

coltrane

he/him

CreditAttribution: coltrane commented 13 September 2012 at 19:30

File	Size
acquia_search-environment-delete-1778282-3.patch	1.32 KB

I can confirm the issue and patch in #2 fixes the read-only problem and doesn't delete the environment. However, I just want to confirm that it's OK that search isn't available during this time. A search fails with "The Apache Solr search engine is not available."

Also, patch in #2 calls acquia_agent_get_subscription() which is first invoked in acquia_agent_check_subscription() during the cron call, so this can result in an extra HTTP request just to get the subscription integer. Instead lets pass that int onto the acquia_subscription_status hook like the attached patch.

Chris, I assume this is also a problem in Drupal 6?

Comment #4

cpliakas CreditAttribution: cpliakas commented 13 September 2012 at 19:43

Thanks for reviewing! Yes, the server being unavailable is OK, but my testing has shown that it isn't given that search has the appropriate credentials stored from previos interactions with rpc.acquia.com. If that is not happening then it is an issue, but to me it is a follow-up issue that we can work though later.

Re: Drupal 6, it is a problem, but a slightly different one. If the same code exists then I believe this fix can be backported, although I haven't tested that yet.

Also, removing the extra http request is a good thing, great catch.

Going through one final test right now, will post back soon.

Thanks,
Chris

Comment #5

cpliakas CreditAttribution: cpliakas commented 13 September 2012 at 20:51

Status:

Needs review

» Reviewed & tested by the community

Tested out the patch, and it works exactly as you described. Unfortunately I was wrong in that search is still unavailable during this time as you said, but at least the environment is not deleted so that it can come back up automatically on the next successful cron run.

Marking as RTBC since it is the most critical to get the deleting get resolved, but I will post a separate issue where we can iterate over search being unavailable after rpc.acquia.com cannot be connected to. The problem is that the _acquia_search_derived_key() function required the subscription data to generate the derived key. Because the subscription data is not available the derived key cannot be calculated and the search request fails. Anyways, that is a trickier issue which will require some thought in order to resolve.

Thanks,
Chris