By default, Solr Spatial queries with polygons not work with error like Unable to parse shape given formats \"lat,lon\", \"x y\" or as WKT because java.text.ParseException: java.lang.UnsupportedOperationException: Unsupported shape of this SpatialContext. Try JTS or Geo3D..

But simple geofit queries like {!geofilt pt=56.325773,44.016162 sfield=rpts_field_geolocation d=10}works well without errors.

Example of queries that not works:

  • fq=rpts_field_geolocation:"Intersects(-74.093 41.042 -69.347 44.558)"
  • fq=rpts_field_geolocation:"IsWithin(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30))) distErrPct=0"
  • fq=rpts_field_geolocation:"IsDisjointTo(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30))) distErrPct=0"
  • fq=rpts_field_geolocation:"Contains(POLYGON((-10 30, -40 40, -10 -20, 40 20, 0 0, -10 30))) distErrPct=0"

For solve this problem we need, at fist, add JTS library to Solr core, as described here: https://lucene.apache.org/solr/guide/8_1/spatial-search.html#jts-and-pol...

So I download file, place it to /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/jts-core-1.15.0.jar, and restart Solr core.

Next step is add JTS factory to our fieldType, I solve this via manual editing schema.xml file, so I find field location_rpt definition:

    <!-- An alternative geospatial field type new to Solr 4.  It supports multiValued and polygon shapes.
      For more information about this and other Spatial fields new to Solr 4, see:
      http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4
    -->
    <fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
        geo="true" distErrPct="0.025" maxDistErr="0.001" distanceUnits="kilometers" />

and extend it via new attributes (maybe some new attributes are not necessary):

    <fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
        spatialContextFactory="JTS"
        autoIndex="true"
        validationRule="repairBuffer0"
        distErrPct="0.025" maxDistErr="0.001" distanceUnits="kilometers" />

After this I recreate index via 'Clear all indexed data' on Search API index Drupal page, reindex entities, and those queries starts to work nomally without errors!

But the problem in search_api_solr side, that those attributes will be removed when we modify index configuration, because module will rebuild schema.xml from scratch.

Solution will be add those attributes to schema.xml generator in module. Can you add those attributes by default, or, maybe, optionally? Thanks!

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Murz created an issue. See original summary.

Murz’s picture

Issue summary: View changes
aleix’s picture

I think that the smart way to allow jts is to create a new SearchApiBackend extending some existing one. Creating the relevant form checkbox to enable/disable the option as you said.
But also is possible to alter all backends schema.xml output...You can just do it using the hook hook_search_api_solr_config_files_alter as here:

/**                                                                                             
 * Alter the newly assembled Solr configuration files.                                          
 *                                                                                              
 * @param string[] $files                                                                       
 *   Array of config files keyed by file names.                                                 
 * @param string $lucene_match_version                                                          
 *   Lucene (Solr) minor version string.                                                        
 * @param string $server_id                                                                     
 *   Optional Search API server id. Will be set in most cases but might be                      
 *   empty when the config generation is triggered via UI or drush.                             
 */                                                                                             
function your_module_search_api_solr_config_files_alter(array &$files, string $lucene_match_version, string $server_id = '') {
  $jts_spatial = '<fieldType name="location_rpt" class="solr.SpatialRecursivePrefixTreeFieldType" spatialContextFactory="org.locationtech.spatial4j.context.jts.JtsSpatialContextFactory"          
autoIndex="true" validationRule="repairBuffer0" distErrPct="0.025" maxDistErr="0.001" distanceUnits="kilometers" />';                                                                              
  $files['schema.xml'] = preg_replace("@<fieldType\s+name=\"location_rpt\"\s+class=\"solr\.SpatialRecursivePrefixTreeFieldType\"\s+geo=\"true\"\s+distErrPct=\"0\.025\"\s+maxDistErr=\"0\.001\"\s+ 
distanceUnits=\"kilometers\"\s+/>@ms", $jts_spatial, $files['schema.xml']);
}                                                                                                                                                                                                  
mkalkbrenner’s picture

I think that the smart way to allow jts is to create a new SearchApiBackend extending some existing one. Creating the relevant form checkbox to enable/disable the option as you said.
But also is possible to alter all backends schema.xml output...You can just do it using the hook hook_search_api_solr_config_files_alter as here:

I don't agree that extending SearchApiBackend would be a smart choice ;-)

The hook in #3 is the way to go unless we have all field types configurable like the text based field types.

aleix’s picture

oops... ok! So the patch I am attaching is not the way to go?... So do you think that it is better to use the hook as in #3 ?

In conjunction with a geofield it's working fine. So I'm able to store a polygon. And then with a given coordinates point I can search for the polygons that are containing that point.

thank's !

aleix’s picture

while not reviewing what I said in the previous patch.
I have created a module that adds the configuration as #3 explains, also, it adds more features such as a views filter plugin to do JTS queries , by now just "contains" predicate can be used. Also modifies the rpt datatype provided by search_api_location to allow index shape fields. So this way one can lookup for shapes that are containing a given lat,lon point.
I hope you can review and found it as useful as I do... it's in https://www.drupal.org/project/jts_solr_queries

mkalkbrenner’s picture

Version: 8.x-3.x-dev » 4.x-dev
Murz’s picture

Status: Active » Needs review
FileSize
2.62 KB

On my sites with this patch all works well long time. Here is re-rolled patch to match last version from git. Please review it, and if all is right, maybe commit it to module core?

mkalkbrenner’s picture

Status: Needs review » Needs work

The patch looks basically good. But it requires more work and will not pass the tests as it is!

  1. +++ b/src/Plugin/SolrConnector/StandardSolrConnector.php
    @@ -50,5 +50,10 @@ class StandardSolrConnector extends SolrConnectorPluginBase {
    +      $files['schema.xml'] = preg_replace("@<fieldType\s+name=\"location_rpt\"\s+class=\"solr\.SpatialRecursivePrefixTreeFieldType\"\s+geo=\"true\"\s+distErrPct=\"0\.025\"\s+maxDistErr=\"0\.001\"\s+distanceUnits=\"kilometers\"\s+/>@ms", $jts_spatial, $files['schema.xml']);
    

    The regex should be less specific, for example
    <fieldType[^>]+name=\"location_rpt\"[^>]*/>

  2. +++ b/src/SolrConnector/SolrConnectorPluginBase.php
    @@ -107,6 +107,7 @@ abstract class SolrConnectorPluginBase extends ConfigurablePluginBase implements
    +      'jts' => FALSE,
    

    "jts" needs to be added to config/schema/search_api_solr.connector.standard.schema.yml

  3. The existing configs for the test servers need to be updated to contain the "jts" flag.
  4. At least the config generator test has to be extended, following the pattern of "jmx".
  5. An integration test would be nice. At least we should ensure that Solr starts without errors.
  6. We need to verify which Solr versions can handle this. Maybe some adjustments to the search_api_solr_legacy module is required.

You should search the entire module for "jmx". This will give you a good orientation what needs to done!

Murz’s picture

@mkalkbrenner, thanks for detailed description!
Here is updated patch, where I simplify regex, and add "jts" to schemas.
Todo:
1. Make config generator test
2. Try to make integration test.
3. Try to find way for check availability of "jts" on server.

aleix’s picture

Nice to see this issue ressurection :) Despite I am using an alter to carry on the jts support as we talk in previous comments( and everything is running fine). Will help with the above when I have time .

mkalkbrenner’s picture

The patch in #10 looks much better :-)

@aleix If you find some time, it would be nice if you can complete the test part. Searching the tests folder for "jmx" will give you a good starting point.

BramDriesen made their first commit to this issue’s fork.

BramDriesen’s picture

I rebased the patch of #10 in a fresh issue fork from 4.x-dev. This will make it easier to rebase in the future if needed and to continue working on this.

Murz’s picture

Status: Needs work » Needs review

@BramDriesen, thanks, rebased patch from https://git.drupalcode.org/project/search_api_solr/-/merge_requests/4 works well!

BramDriesen’s picture

Status: Needs review » Needs work

As per #12 we still need tests :-)

mkalkbrenner’s picture

Yes, I'll merge it as soon as there's a test. As mentioned in #12 it should be straight forward.

mkalkbrenner’s picture

Status: Needs work » Needs review
FileSize
11.13 KB

The patch contained some small errors as it altered the schema.xml only for those Connectors that inherit the StandardSolrConnector. But the config itself is part of SolrConnectorPluginBase.

I also added the test and took care of the Solr36Connector.

  • mkalkbrenner committed 340c7b4 on 4.x
    Issue #3072895 by BramDriesen, Murz, aleix, mkalkbrenner: Add support...
  • mkalkbrenner committed 5f880be on 4.x
    Issue #3072895 by BramDriesen, Murz, aleix, mkalkbrenner: Add support...
mkalkbrenner’s picture

Status: Needs review » Fixed
BramDriesen’s picture

Good job @mkalkbrenner!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.