I was wondering if there has been any documentation or if others have found ways to index and search(view) external content. Currently I have external content indexed into Solr via the data import handler and have mimic my drupal content (field names etc). But I am failing to see any external content being displayed in a search view. The count of results accurately displays the both the internal and external content but the display only shows internal documents. I believe its being thrown out because the search api is trying to do something to verify that all the content types were loaded cause I a seeing a warning from the below code:
loadItemsMultiple(array $item_ids) {
// Group the requested items by datasource. This will also later be used to
// determine whether all items were loaded successfully.
$items_by_datasource = array();
foreach ($item_ids as $item_id) {
list($datasource_id, $raw_id) = Utility::splitCombinedId($item_id);
$items_by_datasource[$datasource_id][$raw_id] = $item_id;
}
// Load the items from the datasources and keep track of which were
// successfully retrieved.
$items = array();
foreach ($items_by_datasource as $datasource_id => $raw_ids) {
try {
$datasource = $this->getDatasource($datasource_id);
$datasource_items = $datasource->loadMultiple(array_keys($raw_ids));
foreach ($datasource_items as $raw_id => $item) {
$id = $raw_ids[$raw_id];
$items[$id] = $item;
// Remember that we successfully loaded this item.
unset($items_by_datasource[$datasource_id][$raw_id]);
}
}
catch (SearchApiException $e) {
watchdog_exception('search_api', $e);
// If the complete datasource could not be loaded, don't report all its
// individual requested items as missing.
unset($items_by_datasource[$datasource_id]);
}
}
// Check whether there are requested items that couldn't be loaded.
$items_by_datasource = array_filter($items_by_datasource);
if ($items_by_datasource) {
// Extract the second-level values of the two-dimensional array (that is,
// the combined item IDs) and log a warning reporting their absence.
$missing_ids = array_reduce(array_map('array_values', $items_by_datasource), 'array_merge', array());
$args['%index'] = $this->label();
$args['@items'] = '"' . implode('", "', $missing_ids) . '"';
\Drupal::service('logger.channel.search_api')
->warning('Could not load the following items on index %index: @items.', $args);
// Also remove those items from tracking so we don't keep trying to load
// them.
// foreach ($items_by_datasource as $datasource_id => $raw_ids) {
// $this->trackItemsDeleted($datasource_id, array_keys($raw_ids));
// }
}
// Return the loaded items.
return $items;
}
Any help or tips on this would be great. The only things I was able to find are the following, which I am not sure are valid anymore:
https://www.drupal.org/node/2717589
https://www.drupal.org/node/2682347
Comments
Comment #2
ndrake86 commentedSo far, this is what I have in hopes this might help some else start and continue building an external datasource plugin for the search API. The properties are specific for what I was accessing but I do think they can be generalized. The problem I am currently having is the search API tries to load the original entity which of course doesn't exist in any drupal known datasource. Extending the
ComplexDataTypeand setting theoriginalEntitywould was the recommendation I found but I am little stuck on how to do that successfully.code so far
Comment #3
ndrake86 commentedComment #4
ndrake86 commentedThis is the source from which I started. Its rather old but is how I got this started -> https://www.drupal.org/node/2701729
Comment #5
ndrake86 commentedComment #6
drunken monkeyMaybe create your own class implementing
ComplexDataInterface, wrap your results data in that and then set it as the original item in the Search API result items? That should do it.Just providing all necessary fields' values in the result items should also work, but there might also be bugs regarding that.
Comment #7
ndrake86 commented@drunken monkey thanks for the reply, I had a feeling (based on some of the other similar issues) that extending the complexDataType class would be a potential solution. I did tried the second suggestion providing all the fields in the data set but it would always fail at getOriginalObject call or some reference to a search API specific field. There is also an issue with access checking that I am not sure how I can solve. Mainly wanted to comingle both external and internal (Drupal made) content. Which based on what I am seeing for access checks would also need to be taken care of. I am trying to work through extending the complexDataType but have been running into issues there. Continuing to work through. Again thanks for the suggestions, hopefully I can get this working at least in my specific case then can throw it back out to become more generalized.
Comment #8
cato commentedSo @ndrake86 did you get it working? I'm doing something similar where we want to query an external index.
Comment #9
dcam commentedI've been working on a Solr datasource module for the past several weeks.* The sandbox project can be found on my GitHub repo. This project should be considered pre-alpha, unstable, and unsecure. Please only set it up in a development environment.
I have recently been successful in displaying external Solr data in Drupal using this module. Differences in server configurations and Search API index configurations may cause issues with displaying your data. Information about setting up the module, warnings about using it, and a list of known issues are all included in the README file.
*I didn't find this issue when I looked for info about datasource creation. It is gratifying to see @drunken_monkey recommend the same procedure for developing it that I figured out on my own.
Comment #10
mkalkbrennerI mark this issue as fixed to clean up the queue.
BTW I would love to see dcam's module as a sandbox project on drupal.org.
Comment #11
dcam commentedHere you go: https://www.drupal.org/sandbox/dcam/2867828.
Comment #12
mkalkbrennergreat!
Comment #13
drunken monkeyFeel free to mark as "fixed" again if you're sure, but I think this issue should remain open, with the goal of eventually getting this functionality built-in as part of this module (or at least as a sub-module in this project).
Comment #14
mkalkbrennerhttps://www.drupal.org/node/2618414#comment-12052047
It seems like dcam development goes in the right direction. I like the idea to have that additional module for that use-case.
Comment #15
mkalkbrenner