Attachments from past revisions of parent appearing in index

Could you write up a scenario how someone can test this bug?

It feels like there is indeed something wrong and I'd love to give it a try but I'm hoping for a clear tutorial to show me how I can see things are wrong.

Thanks!

I'm experiencing the same issue. Files used in previous revisions of a node are coming up in the search results. If you have enabled Revisions in your content-type, and have a field of type "File", you can run this test:

1. Find or create a node that has no file associated with it.
2. Upload a PDF that contains a unique word and save the node.
3. Index queued content at admin/config/search/apachesolr.
4. Search on the unique word to make sure the PDF is being indexed.
5. Remove the file from the node and Save.
6. Re-index queued content (and clear any necessary caches) to make sure search index is fresh.
7. Search on the unique word. It will show up and point to the PDF that was saved with the revision.

It seems to me this is also a problem of Revisioning, which retains old files (re issue 1816584) at their original paths, but really, Apache Solr Attachments should remove the contents of files that have been revisioned from the index.

We are seeing the exact same issue here...

I have the issue as well, is any solution now?

I was able to resolve this on my site but your mileage may vary. When you look at the Configuration settings on the Apache Solr Default Index tab (/admin/config/search/apachesolr), you might try unchecking anything under FILE. Then go over to the Attachments tab and open the Bundle configuration page. There you can specify that your file attachments are searched as part of the parent entity of the content-type you wish to have searchable. That is enough to get the contents of your files indexed, but only if they are currently attached to the node. At least, when I ran the test I described above, the PDF disappeared from my search results after I removed it from the node.

The consequence of this, however, is that if you have any files that are not specifically attached to a node through a dedicated file field, they will not be indexed in search. But this may be a good thing depending on your workflow.

I have the same problem and I need to keep attachments being indexed as standalone objects. Has anybody an additional idea on how to solve this problem?

Anybody??

Hi Drupal Community,

I'd like to share with you my solution for this headache. The idea is to add new field to Solr index for `file` entity_type `bs_latest_revision` that defines if the file comes from latest revision. Then it's just a matter of query alter based on this field in order to retrieve files only from latest revision. Here is ready solution (improvements are nicely welcome):

1. Implement hook_entity_info_alter.

/**
 * Implements hook_entity_info_alter().
 */
function YOURcustomMODULE_apachesolr_entity_info_alter(&$entity_info) {
  $entity_info['node']['document callback'][] = '_YOURcustomMODULE_solr_document';
}

2. Implement helper function to extend `file` entity type in Solr index.

/**
 * ApacheSolr Attachments Extension
 */
function _YOURcustomMODULE_solr_document(ApacheSolrDocument $filedocument, $parent_entity, $parent_entity_type, $env_id) {
  if (!empty($filedocument->entity_type) && !empty($parent_entity_type) && $filedocument->entity_type == 'file' && $parent_entity_type == 'node') {
    // Add info if the file comes from most recent revision.
    $in_latest_revision = 0;
    list($parent_entity_id, $parent_entity_vid, $parent_entity_bundle) = entity_extract_ids('node', $parent_entity);
    $fields = field_info_field_by_ids();
    if (is_array($fields)) {
      foreach ($fields as $field_id => $field_info) {
        if ($field_info['type'] == 'file') {
          foreach ($field_info['bundles'] as $entity_type => $bundles) {
            if (in_array($parent_entity_bundle, $bundles)) {
              $file_field_names[$field_info['field_name']] = $field_info['field_name'];
            }
          }
        }
      }
    }
    foreach ($file_field_names as $file_field) {
      if (isset($parent_entity->$file_field)) {
        $parent_entity_file_fields = $parent_entity->$file_field;
        //@todo deal with different languages properly
        foreach($parent_entity_file_fields as $language => $files) {
          foreach ($files as $k => $file) {
            $file = (object)$file;
            if ($file->fid == $filedocument->entity_id) {
              $in_latest_revision = 1;
            }
            break 3;
          }
        }
      }
    }
    $filedocument->setMultiValue('bs_latest_revision', $in_latest_revision);
  }
  return array(); // all alterations are made to $document passed in by reference
}

3. Drupal search query alteration

/**
 * Implements hook_apachesolr_query_prepare().
 */
function YOURcustomMODULE_apachesolr_query_prepare(DrupalSolrQueryInterface $query) {
  $filter = new SolrFilterSubQuery('AND');
  $sub_q = new SolrFilterSubQuery();
  // OR operator is used by default!
  $sub_q->addFilter('bs_latest_revision', true);
  $sub_q->addFilter('entity_type', 'node');
  // Apply the query!
  $filter->addFilterSubQuery($sub_q);
  $query->addFilterSubQuery($filter);
}

4. Clear Cache

5. Reindex.

Hope it helps,
Peace! Michał K.

@michalk: nice solution!! Just one feedback to your code: seems that $filedocument->setMultiValue (in _YOURcustomMODULE_solr_document) is deprecated. Can we use $filedocument->addField instead?
And, how are your index settings? Did you enable "File->Document" on "Default index" page?
And, what are your bundle settings on the "Attachments" page? I guess you set "Attachments as separate entities" for each content type? What about the "Document" bundle? Did you enable indexing here or did you choose "Don't index attachments" instead?

@michalk: tried your code - but now nothing is in the search results anymore. No nodes, no attachments, no users ...
Problem seems to be with the YOURcustomMODULE_apachesolr_query_prepare function - if I comment this function, search results are shown again.

Hi @vistree,

I just tested the code against my website and it works fine. Have you cleared the cache and reindexed?

And yes "File->Document" enabled.
And yes "Attachments as separate entities" for each content type.

With the current condition you wont get the users in search results, I believe that if you add this it will give you users as well:
$sub_q->addFilter('entity_type', 'user');

Maybe your Solr engine version is the problem. I use 5.5.5

Check if "bs_latest_revision" field has been indexed by Solr.

Please let me know if you got any progress.
Kind regards,
M.K.

Hi Michal,
still investigating to get your workaround running. Seems, that the bs_latest_revision field is indexed by SOLR (bs_latest_revision boolean 2) - but I need to get access to the solr itself to check engine version and to check the index itself.
Will provide feedback as soon as possible ;-)

Hi Michal,
should the field bs_latest_revision exist on file entity only or also on the node entity? Installed solr_devel module an can get information for node like this:

id (String, 18 characters ) XXXXX/node/197536
site (String, 29 characters ) https://www.devel.local
hash (String, 6 characters ) XXXXX
entity_id (Integer) 197536
entity_type (String, 4 characters ) node
bundle (String, 10 characters ) searchtest
bundle_name (String, 10 characters ) searchtest
ss_language (String, 2 characters ) en
path (String, 11 characters ) node/197536
url (String, 69 characters ) https://www.devel.local/en/search-test-onesh...
path_alias (String, 39 characters ) search-test-brochures-all-2018
label (String, 40 characters ) Search test: brochures all 2018
content (String, 1407 characters ) Through the share purchase p...
teaser (String, 302 characters ) Through the share purchase p...
ss_name (String, 28 characters ) XXXXX@devel.local
tos_name (String, 28 characters ) XXXXX@devel.local
ss_name_formatted (String, 12 characters ) My Name
tos_name_formatted (String, 12 characters ) My Name
is_uid (Integer) 1246
bs_status (Boolean) TRUE
bs_sticky (Boolean) FALSE
bs_promote (Boolean) FALSE
is_tnid (Integer) 0
bs_translate (Boolean) FALSE
ds_created (String, 20 characters ) 2020-03-04T12:38:00Z
ds_changed (String, 20 characters ) 2020-08-01T10:03:33Z
ds_last_comment_or_change (String, 20 characters ) 2020-08-01T10:03:33Z
timestamp (String, 24 characters ) 2020-08-01T11:02:55.404Z
sm_og_group_ref (Array, 1 element)
spell (Array, 2 elements) 

Initial test gives me the suspicion that the code in #10 sort of requires revisions. The parent node without revisions should produce file SOLR index records that always have "bs_latest_revision": true, but I see here that for one of the two attached files "bs_latest_revision": false. With SOLR queries I could find more attachments indexed with bs_latest_revision false of parent nodes that had no revisions . I will try to resync my data from production re-do indexing to see if I can reproduce the same behavior. Depending on my client's priorities I might also provide alternative code.

Can someone who better understands the code confirm my suspicion?