Problem/Motivation

Enabling modules Search Api Solr, Search API Location & Geofield modules, I'd like to index the location Lat/Lon pair stored in geo field.

However, it's not possible because the location field is not supported by SearchAPI Solr Backend & Location DataType is not marked a default because it's not supported by all the backends as others like decimal or text.

Proposed resolution

Implement method supportsDataType() in SearchApiSolrBackend class to mark this datatype as supported by Solr indexes.

Remaining tasks

User interface changes

None

API changes

Implementation of new method in SearchApiSolrBackend class.

Data model changes

None

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

plopesc created an issue. See original summary.

plopesc’s picture

Title: Add location as a supported DAtaType for SearchApiSolrBackend » Add location as a supported DataType for SearchApiSolrBackend
Status: Active » Needs review
FileSize
844 bytes

Attaching patch.

Thanks

plopesc’s picture

FileSize
847 bytes

Wrong patch, review this last one, please.

Sorry for the noise.

ndrake86’s picture

From my experience even if the unsupported box is checked a field with that type will get it into your index as long as you have the correct dynamic field definition defined in the schema.xml.

For me I created a custom ngram field definition:

* Provides a full text data type.
 *
 * @SearchApiDataType(
 *   id = "ngram",
 *   label = @Translation("Ngram Fulltext"),
 *   description = @Translation("Custom full text field with edgeNgramFilter with no stemming custom tokenizer."),
 *   prefix = "ng"
 * )
 */

It shows that its unsupported, but its not, but the issue I was having is that all fields currently in the search_api_solr go through a check to see if they are single value fields or multivalue fields and take on a "s" or "m" respectively. So in my schema.xml adding two dynamic fields ngm_* and ngs_* result in the data being indexed correctly.

So the utilities.php file has the prefix as *loc* so is probably is coming through as something like locm_* or locs_* which are both in the schema file I have:

<dynamicField name="locs_*" type="location" indexed="true"  stored="true" multiValued="false"/>
 <dynamicField name="locm_*" type="location" indexed="true"  stored="true" multiValued="true"/>

did you query solr directly to see if it actually got indexed?

mkalkbrenner’s picture

mkalkbrenner’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

As discussed at the Drupal Dev Days Milan 2016, we should add at least an integration test like we did for factes.
And we need to check if it still works when #2759205: escapeFieldName() is obsolte gets applied because that escaping especially targeted the location field.

mkalkbrenner’s picture

ndrake86’s picture

I think this patch is still valid, it doesn't look like the search_api modified public function supportsDataType($type) { just the supportsFeature() method.

mglaman’s picture

Status: Needs work » Needs review
FileSize
1.15 KB

Here is a re-roll. Going to test.

mglaman’s picture

With this patch, and using my personal D8 sandbox port of GeoIP (https://github.com/mglaman/geoip) I can filter by location distance.

  public function query() {
    /** @var \Drupal\geoip\Plugin\GeoLocator\Local $local */
    $local = $this->geoLocatorManager->createInstance('local');

    $ip = \Drupal::request()->getClientIp();
    if ($ip == '127.0.0.1') {
      $ip = '162.243.241.142';
    }
    /** @var \GeoIp2\Model\City $record */
    $record = $local->getRecord($ip);

    $location_options = (array) $this->query->getOption('search_api_location', []);
    $location_options[] = [
      'field' => $this->realField,
      'lat' => $record->location->latitude,
      'lon' => $record->location->longitude,
      'radius' => reset($this->value),
    ];
    $this->query->setOption('search_api_location', $location_options);
  }
mollux’s picture

I worked on this during Ironcamp, and I got a working solution that is also able to return the distance, and sort the results based on the distance.
Together with the latest changes in the 8.x-1.x branch of Search API location module, I was able to make a view with geo filtering and returning the distance from Solr.
See the attached screenshot.

The solution is also more robust as the 7.x branch, as the distance field is linked to the indexed location field, so technically it is possible to index multiple location fields and calculate the distances.

The facets aren't working yet, as the range/slider functionality is blocked on #2611812: Create a date query type

I will continue to work on this the next days and write some text/blogpost to describe the testing process.

mkalkbrenner’s picture

Status: Needs review » Needs work

The patch looks ok, but I need to play with it a little.
Could you (or someone else) provide a small test? Just indexing some locations and firing some queries. I think we need at least this minimal test coverage.

mollux’s picture

I agree we need tests for this.
I can provide a search_api_solr_test_location module in the next days with a test index and some test data + tests.

mollux’s picture

Status: Needs work » Needs review
FileSize
21.8 KB

I worked further on this, and fixed some issues while writing the tests.
I added a dependency on geofield, so the correct indexing of geo data can be tested too.

The location tests currently live in a separate file, as they add overhead not needed for the other solr tests.

dtarc’s picture

This patch allows fields to be indexed as Latitude/Longitude. However, Indexing a field as a Latitude/Longitude breaks the Fulltext search views filter.

I've created an issue in Search API Location, but the issue is probably related to how this patch works.

See #2861350: Indexing a field as Latitude/Longitude breaks Fulltext search views filter.

fran seva’s picture

Hi -- working in #2861350: Indexing a field as Latitude/Longitude breaks Fulltext search views filter I think the problem is caused by this patch because it's adding a server field

+  public function getBackendDefinedFields(IndexInterface $index) {
+    $location_distance_fields = array();
+
+    foreach ($index->getFields() as $field) {
+      if ($field->getType() == 'location') {
+        $distance_field_name = $field->getFieldIdentifier() . '__distance';
+        $distance_field = new Field($index, $distance_field_name);
+        $distance_field->setLabel($field->getLabel() . ' (distance)');
+        $distance_field->setType('decimal');
+        $distance_field->setDatasourceId($field->getDatasourceId());
+        $distance_field->setPropertyPath($distance_field_name);
+
+        $location_distance_fields[$distance_field_name] = $distance_field;
+      }
+    }
+
+    return $location_distance_fields;
+  }

search_api, retrieve all the fields before list the available filters in views using this code:

 foreach ($index-><strong>getFields(TRUE)</strong> as $field_id => $field) {
        $field_alias = _search_api_views_find_field_alias($field_id, $table);
        $field_definition = _search_api_views_get_handlers($field);

it call to getBackendDefinedFields and return a filed that through an exception when we try to get its definition using $field->getDataDefinition() in _search_api_views_get_handlers function.

I think, the error should be handle in search_api_solr but I'm not sure. Any thoughts?

fran seva’s picture

Hi -- I have created #2863736: Create setDataDefinition method in Field class in search_api queue to allow add the field definition to a backend field.
That way the problem will be fixed adding the following line to the _distance backend field

$distance_field->setDataDefinition('decimal');

cbeier’s picture

Status: Needs review » Needs work

Follow up to #17: The setDataDefinition (that is needs to get the fulltext field to work) is now in search_api. But setDataDefinition now expect a item that implements DataDefinitionInterface).

Recoverable fatal error: Argument 1 passed to Drupal\search_api\Item\Field::setDataDefinition() must implement interface Drupal\Core\TypedData\DataDefinitionInterface, string given, called in /search_api_solr/src/Plugin/search_api/backend/SearchApiSolrBackend.php on line 2560 and defined in Drupal\search_api\Item\Field->setDataDefinition() (line 459 of modules/contrib/search_api/src/Item/Field.php).

cbeier’s picture

Status: Needs work » Needs review
FileSize
22.23 KB

Patch attached.

dbjpanda’s picture

Issue summary: View changes
dbjpanda’s picture

Patch applied successfully. It shows green mark for location data type. Data type is now supported. But I am getting an exception while trying to index data.
Drupal\search_api\SearchApiException while trying to index items on index Default Solr content index: Solr HTTP error: OK (400) {"responseHeader":{"status":400,"QTime":253},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.locationtech.spatial4j.exception.InvalidShapeException"],"msg":"Point must be in 'lat, lon' or 'x y' format: Bhubaneswar","code":400}} in Drupal\search_api_solr\Plugin\search_api\backend\SearchApiSolrBackend->indexItems() (line 711 of C:\Users\Dibya\Sites\devdesktop\drupal-8.2.6\modules\search_api_solr\src\Plugin\search_api\backend\SearchApiSolrBackend.php)

dbjpanda’s picture

Status: Needs review » Needs work
cbeier’s picture

Rerolled patch.

dbjpanda’s picture

Assigned: Unassigned » dbjpanda
Status: Needs work » Needs review

Going to review it soon.

dbjpanda’s picture

Status: Needs review » Needs work

Getting these errors while applying the patch #23
Solarium\Exception\HttpException while indexing: Solr HTTP error: OK (400) {"responseHeader":{"status":400,"QTime":5},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.locationtech.spatial4j.exception.InvalidShapeException"],"msg":"Point must be in 'lat, lon' or 'x y' format: Delhi","code":400}} in Solarium\Core\Query\Result\Result->__construct() (line 106 of C:\Users\Dibya\Sites\devdesktop\drupal-8.2.6\vendor\solarium\solarium\library\Solarium\Core\Query\Result\Result.php).

Drupal\search_api\SearchApiException while trying to index items on index Default Solr content index: Solr HTTP error: OK (400) {"responseHeader":{"status":400,"QTime":5},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.locationtech.spatial4j.exception.InvalidShapeException"],"msg":"Point must be in 'lat, lon' or 'x y' format: Delhi","code":400}} in Drupal\search_api_solr\Plugin\search_api\backend\SearchApiSolrBackend->indexItems() (line 708 of C:\Users\Dibya\Sites\devdesktop\drupal-8.2.6\modules\search_api_solr\src\Plugin\search_api\backend\SearchApiSolrBackend.php)

gvigroux’s picture

The patch works for me! I applied it on top of the Beta 3 (not working on top of the dev version).
Now I can index data.

The first time the indexation failed because values where not in range -180;180 and -90;90

mkalkbrenner’s picture

Assigned: dbjpanda » Unassigned

@mollux
Can you review the latest patch?

dbjpanda’s picture

@gvigroux Thanks. I will test it against the Beta 3 soon. And @mollux Kindly review if you have time.

gvigroux’s picture

I have tested the patch 23 with the "search_api_solr" RC1 but it's not working due to file changes

mkalkbrenner’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests, -drupalironcamp2016
FileSize
21.78 KB
22.68 KB

I reviewed and adjusted the patch. The tests pass locally.

  • mkalkbrenner committed 5f00880 on 8.x-1.x authored by mollux
    Issue #2744733 by plopesc, mollux, cbeier, mglaman, mkalkbrenner, fran...
mkalkbrenner’s picture

Status: Needs review » Fixed

At least on PHP 7 all tests pass on travis :-)
https://travis-ci.org/mkalkbrenner/search_api_solr/builds/231185981

Due to the fact that we already got the feedback here that "the patch works" I don't won't to block the feature any longer and created a follow-up:
#2877718: Location tests fail on PHP 5

Thanks everybody for your help!

gvigroux’s picture

Thanks for the update :)
I have updated my project and I can still index "Latitude/Longitude" data types.

I have an issue to do a proximity search but I guess the problem is in "Search API Location" now ;)

gvigroux’s picture

Status: Fixed » Needs review

Comming back on this thread because I'm not sure if the indexed filter is usable.
I have opened a ticket in search_api_location: https://www.drupal.org/node/2879632 but the schema problem should maybe be handled here.
Am I right?

mkalkbrenner’s picture

Status: Needs review » Fixed

Please don't re-open this issue. The topic here is fixed!

If required, open a new issue. But if you talk about this error message, this one is not related to solr in any way:

InvalidArgumentException: The configuration property display.default.display_options.filters.latlon.value.distance doesn't exist. in
Drupal\Core\Config\Schema\ArrayElement->get() (line 74 of core\lib\Drupal\Core\Config\Schema\ArrayElement.php).

This message is about a drupal config schema.

gvigroux’s picture

Sorry for that! Understood

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.