Problem/Motivation
I store Geofield multipolygon values in Solr (as type "Storage-only") and don't use the "Use entity field rendering" feature in Search API Views integration for better performance. (See : Create a search view that doesn’t load entities from the database)
I display the multipolygons with Leaflet and do get an output, but only points are shown, not multipolygons. If I switch to "Use entity field rendering" the multipolygons are rendered fine.
Steps to reproduce
- Setup a content type with a Geofield field, and enter multipolygon values
- Setup Search API with Solr and index the Geofield field as type Storage
- Create a View with the Solr index
- Add a Geofield field. Ensure that "Use entity field rendering" is unchecked
- Set view format to "Leaflet Map"
- Save and view the page, and see that only points are shown
Workaround
Note: This defeats the purpose of storing values in Solr, since the values are then collected in the slow fashion, from the database.
- Set the Geofield field to "Use entity field rendering" > Formatter: Raw Output, Output format: WKT
- See that the multipolygons are showing
Proposed resolution
It seems like Search API, Geofield or Leaflet grabs only the first lat/long value from the polygon values, and uses that as a point. It would be fantastic if Search API/Geofield/Leaflet was able to auto-detect the format of the geo-location values, and make a qualified guess whether it is Point, Multipolygon, etc., and set the formatter to the best matching format. had geofield_default ("Raw Output") defined as a fallback FieldFormatter using the $fallbackHandler function in Search API, since I assume the polygons would show, not just points.
- $fallbackHandler in Search API
- $fallbackHandler example implementation in Elasticsearch Connector module
- Geofield's Plugin implementation of the 'geofield_default' formatter
- Leaflet Map Views integration << Set fallback to
geofield_defaultvia $fallbackHandler
Alternatively, using a simple hook in a custom module would be a totally acceptable workaround.
Remaining tasks
- Document how to define the desired renderer in a custom module with a hook (easier?)
- -- OR --
- Enhance the code to use
geofield_default("Raw Output") via the Search API$fallbackHandler
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #52 | 3372686-52.patch | 911 bytes | idebr |
| #15 | 3372686-14 2.2 Without Entity rendering .png | 129.68 KB | ressa |
| #15 | 3372686-14 2.1 With Entity rendering .png | 103.9 KB | ressa |
| #15 | 3372686-14 1 breakpoint and row.png | 185.37 KB | ressa |
| #14 | leaflet-view-solr-2023-9-9.tar_.gz | 1.28 MB | ressa |
Comments
Comment #2
ressaThis should probably be a feature request, since it is not a bug, and the module works well for normal use.
Comment #3
ressaMoving to Search API module, since that is probably a more correct place in the process to address this?
Comment #4
ressaMoving back to Leaflet, since I discovered the
$fallbackHandlerfunction in Search API.Comment #5
ressaComment #6
ressaAnd now I am moving it back to Leaflet :)
Comment #7
ressaSpecify that
geofield_default("Raw Output") is the FieldFormatter in play.Comment #8
itamair commentedThanks for opening this Ressa.
But it would be pretty complicate for me to setup and reproduce your use case for me ...
both because I don't have time to dedicate to this, also not real scenario to apply and also not so familiar with Solr (can you believe I really never setup it up locally, etc. ...?).
Would you (be able to) go on, on this, and provide direct contribution ... such as
proper debugging and fix of this, throughout a (strictly not regressive and PHP/Drupal coding standard compliant) patch or Merge Request ...?
Comment #9
ressaI understand @itamair, there are many components in play here ... Sadly, my coding skills are too limited to update the code.
If you're interested, I could create a very simple example for you, and include the steps to set up and import it into a DDEV/Solr instance? Setting up Solr in DDEV is super easy: https://github.com/ddev/ddev-drupal9-solr/
Alternatively, if you don't have time right now, do you think it's possible to use a simple hook in a custom module to specify that
geofield_default("Raw Output") should be used as FieldFormatter?Comment #10
itamair commentedWell of course it would help if you create an example to look at, or to clone and reproduce locally with DDEV.
If you help me reproducing your use case I could look into it and better understand if the simple hook or whatever could be the most appropriate solution ...
Comment #11
ressaThanks, that sounds good. I tried to replicate the problem, and the other day it worked as expected, and I couldn't replicate it. But then I did the same steps (maybe with other field names?) and the issue was back...
So here are Composer files and database dump of a very basic site for import into DDEV, containing two nodes with polygons stored in Solr, and a Views map. Let me know if anything is unclear in the README.md about setting it up.
Comment #12
ressaAlso, the title might not be correct, since it seems like sometimes, the polygons do show with "Use entity field rendering" disabled, so fallback might actually work ... but I won't update it, since I am not sure.
Comment #13
ressaPlease let me know if you need more info.
Comment #14
ressaI have updated the modules, and added a step to update Solr configuration in DDEV, which I might have forgotten in the first set up from July 2023. Sorry about that. I am attaching a fresh tar.gz-file with updated Composer files, database dump, etc. to allow restoring a local example Drupal 10 install with Leaflet and Solr in DDEV.
I finally got Xdebug breakpoints working in PhpStorm:
I found the place in the code where the problem originates, by adding a breakpoint at line 891 in /modules/leaflet_views/src/Plugin/views/style/LeafletMap.php:
Both with or without "Use entity field rendering" enabled, this is the Views row result, which is correct:
The resulting value in
$geofield_valuelooks like it is the problem:MULTIPOLYGON (((9.387817 56.122081, 8.783569 55.789959, 9.113159 55.542101, 9.981079 55.486117, 10.090942 56.073058, 9.387817 56.122081)), ((9.761353 55.355176, 8.997803 55.429013, 8.794556 55.148535, 9.761353 55.355176)))POINT( 8.783569 55.789959 MULTIPOLYGON (((9.387817 56.122081)It looks like it messes up the format, and mistakenly tries to use the second value as a point, ending up with malformatted value in the end.
PS. drupal.org adds an underscore in the file name, so you need to rename it:
Comment #15
ressaAdding screendumps from PhpStorm.
Breakpoint and Views row value
With Entity rendering, correct value
Without Entity rendering, wrong value returned
Comment #16
ressaIt actually looks like the problem is this function, starting at line 312 in /modules/leaflet_views/src/Plugin/views/style/LeafletMap.php:
If the "Use entity field rendering" is disabled, the Geofield value skips the
[$lat, $lon] = explodepart and is in the correct format ("MULTIPOLYGON (((12.7342 55.7032,12.7339 55.7027,12.7334 [...]") and the multipolygons are rendered correctly.With "Use entity field rendering" disabled, the value is exploded into
$lat$lonvalues, like this:@itamair: Should we reach out to @drunken monkey and ask for help?
Comment #17
orkutmuratyilmazHello all,
Actually, new version of Search API solved a lot of problem of mine. Have you tried it with this issue too?
Best,
Orkut
Comment #18
ressaThanks for the suggestion @Orkut Murat Yılmaz, sadly the problem is still there ...
If you are interested in taking a look at this problem, you're more than welcome to check out the Solr example (from 9 September 2023) with Composer files, database dump, etc. to allow restoring a local example Drupal 10 install with Leaflet and Solr in DDEV.
Comment #19
itamair commentedHey Ressa ... sorry for my lack of participation here (but I am much engaged also in some other work stuff, unfortunately not related at all with Drupal Mapping).
I am going to jump on this asap (hopefully in this weekend) and indeed try to reproduce your use case, by using what you provided (also for setting SOLR locally with DDEV).
Will try to come back here with some detailed feedback and possibly also some solutions clues.
Comment #20
ressaHi @itamair, no problem, I totally understand. We all have to prioritize, and paid work comes first, since it is needed to allow voluntary work.
About Getting Paid for Open Source Work, you could consider, as Webform does, to offer "Professional support" or "Fund development" on Geofield and Leaflet via Open Collective.
Anyway, I am very grateful for your efforts with maintaining so many map-related modules in Drupal.
Comment #21
itamair commented@ressa thanks a lot for providing such a solid and detailed testing env on this, with DDEV, Solr and Geofield content already there.
All the configuration and installation process was fine and I was properly able to reproduce and review your use case.
The good news are that it indeed looks something on the Leaflet module side, so we should be able to fix this autonomously.
Could you check the attached patch and properly test / QA and let us know if it solves your issue here? (as I hope ...)
BTW: don't you have any account in the Drupal Slack channel (drupal.slack.com)?
I wanted to live chat with you, on this ... so as on my beloved Denmark
(I lived in your super cool country for 3 splendid years! and I still miss it pretty much)
Comment #22
itamair commentedComment #23
itamair commentedslight (and better) update of the previous patch.
Please focus and QA/Review this one ...
Comment #24
ressaThanks a lot @itamair for looking at this. I am glad that the set up worked well for you, it seemed like the simplest way to share what is a slightly complicated environment.
The patch works great, I can disable "Use entity field rendering" and the Leaflet view is now rendered just as expected, as when "Use entity field rendering" is enabled.
I am not sure how to tell if Solr or MySQL is getting the requests, though ... I tried disabling caching with this:
drush state:set twig_debug 0 --input-format=integer && drush state:set twig_cache_disable 0 --input-format=integer && drush state:set disable_rendered_output_cache_bins 0 --input-format=integer... and then reload the page a few times, by clicking "Admin Toolbar Extra Tools > Flush Views cache", with Webprofiler enabled. These are the results (Solr="Use entity field rendering" disabled):
So there is an improvement, which is great, but do you know if there is a better method to test this? If not, it's fine and I will change status to RTBC, since the error has been fixed by you.
I don't use Slack, sorry. I am glad you enjoyed your time up here in the cold North, where it's now fridge temperature, at 5°C and time for the Long Johns :-) Where did you live?
Comment #25
itamair commentedComment #26
itamair commentedComment #28
itamair commentedHi Ressa ... no, I don't have much experience with Search API and SOLR, and their fine testing.
SOooo ... let's close this as fixed and add this improvement in a brand new Leaflet 10.2.3 release, as all this really looks a well done enhancement to Drupal Leaflet.
PS: I lived 3 great years in the wonderful CPH, working as Senior BE (Drupal) developer for Unity Technologies ...
Comment #29
ressaAll right, since the performance does look good, it can wait until and if it becomes an issue. I tried the latest release, and it works perfectly. Unity, nice -- they became big!
Comment #31
ressaComment #32
lendudeWe updated to 10.2.12 and one map displaying ~600 pins stopped loading, giving out-of-memory errors where previously it was loading fine. Tracked it down to this change. Something here makes the caching go crazy and loopy until
\Drupal\Core\Cache\CacheTagsChecksumTrait::calculateChecksumruns out of memory. Not sure if this is the fault of core caching, search API caching or how this patch changed things but here is a patch reverting this change for people running into similar problems and don't need this functionalityComment #33
ressaThanks for sharing a patch @Lendude -- perhaps an issue should be created, aimed at keeping Geometries support, while fixing the caching issue?
Comment #34
itamair commentedThanks @lendude ...
@ressa, well of course the patch #32 looks reverting all that was done here, so it is not advisable in order to keep Geometries support.
But in my Leaflet playground / testing env I am experiencing much simpler code (keeping the processing simplicity required by @lendude) still able to provide support to Geometries in Search API Leaflet view, and with the option "Use entity field rendering" unchecked.
@resa could you test and QA the new attached patch, that performs a good amount of code simplification, if it is still able to render Geometries in your use case? and also @lendude eventually ...
Comment #35
ressaThanks for a fast patch @itamair, sadly it doesn't work for my map ... I don't get any errors, but the same two nodes are shown, regardless of there being no facets, or after updating facets. I'll try with the attached leaflet-view-solr-2023-9-9.tar_.gz as well.
Comment #36
ressaI have now verified with the example from #14, and it also doesn't work with the patch.
If you follow the steps in the leaflet-view-solr-2023-9-9.tar_.gz README, you should be able to have a working example running in a few minutes. The only thing you need to do is:
DDEV is such an amazing tool.
Comment #37
itamair commented@ressa ... yes, I rebuilt your nice example (provided with the great DDEV) and was able to properly QA & Review patch #34,
and yes indeed we cannot get rid of the specific code under that checks if
$result instanceof SearchApiResultRow(otherwise we would completely loose search API rendering)
BUT it looks that we could further & greatly refactor & simplify (less redundant) the same code in the 10.2.x dev branch at the moment ...
Could you please re-QA and Review the new attached, that looks working fine to me, both in a Search API / Solr Leaflet View and in a normal Leaflet View?
Thaaaanks!
Comment #38
ressaI am glad to report the patch works well, and that the Geofield multipolygon values are rendered fine, and just the same way as with the latest Leaflet release. Also, it renders fine in a regular View as well. Thanks @itamair!
@Lendude: Perhaps you can check if the out-of-memory error is solved with the patch?
Comment #39
lendudeApplied the patch in #37, unfortunately still runs out of memory. No time to test further I'm afraid.
Comment #41
itamair commentedThanks @ressa ... patch #37 is a code improvement and has been committed into 10.2.x branch. It will be part of the next Leaflet module 10.2.x release.
Sorry for your issues #Lendude ... may be you will need to re-roll your patch, to roll back.
But it looks you might be experiencing issues depending by your specific setup.
May be you should provide more extend information on your uses case, how to eventually reproduce it.
And still I am not sure if you are experiencing your out of memory on a normal Leaflet Map View or on a Search API based one
Comment #43
i.koychev commentedHi #Lendude
We have the similar issue when we try to display a list of pins on the leaflet map using Search API, Geofield, Leaflet & Views (the issue is 2 years old Drupal 9/10).
So far we haven't found a solution.
When the pins are more than 800 (we have 1300) the view reports out of memory.
The coordinates in search api index are defined as type of string.
We use Search API Indexed based View with the following settings:
- Format: Leaflet Map
- Data source: coordinates (field)
- Simple Tooltip: title (field)
- Leaflet popup: Decription (field)
- Leaflet Map Tiles Layer: OSM Mapink
- Enable Leaflet Markercluster Js Library functionality - On.
Fields:
- Coordinates
- Title
- Decription
The issue occurs only when we use Search API Index based View.
I'm not sure which module this issue should be addressed to.
Comment #44
ekes commentedJust a heads-up if anyone else notices this. I've not had time to investigate myself but we have a report that this patch broke a Search API based view https://github.com/localgovdrupal/localgov_directories/issues/378
Comment #45
ekes commented> The issue occurs only when we use Search API Index based View.
Regarding the memory usage, as I see it here. I've looked at in the past, and did supply a patch or two that helped bring it down. But what I noted was:
First with Search API either DB or Solr unless you have done quite some work you will be doing an entity load for every row returned (so every point). You can do the work with Solr to make it use the field data and not do an entity load. This will reduce your memory usage significantly. There is a patch in Search API module I started to do the same with the db backend, but it's only a start, and requires much more work.
The second point where there is quite some memory usage is the same if you use views sql or search api, and probably doesn't come into play if you're talking just 1000 points or so. But I'll mention it here as I looked at this too at the same time. When constructing the points, or worse a set of more complicated geometries, for display on the map there is a lot of configuration and the whole loop over all the points and creation of the output is quite intensive. This I don't see any way round, because doing anything else would loose the features of the module, being able to configure the output such. With flexibility comes some complexity.
Comment #46
itamair commentedAgree with @ekes.
This issue changes properly enable Geometries support for Search API Solr and Views integration.
And it is also true that Leaflet View style render method perform some intensive tasks to generate "features" that are then sent to the js layer for FE rendering with the leaflet.js library itself.
Every new logic that could better streamline all those tasks, without breaking any (present or backward) functionality || removing their associated options/properties (that further FE Map alterations could rely upon) are really welcome.
For instance, in most recent commits/versions (https://www.drupal.org/project/leaflet/releases/10.2.15, https://www.drupal.org/project/leaflet/releases/10.2.16) I added token / replacements patterns support for Tooltip and Popup options, for leveraging dynamic use of those (based on features properties) ... those are new cool features, that of course come with (little) additional calculation complexity.
But, worth to mention that (in my actual experience) this complexity needs to be processed only the first time, if proper FE caching is enabled, and after that Leaflet Maps loads really fast, also with hundreds of markers.
Proper tuning of Marekclustering also makes a lot of difference (and performance improvement) in this context.
Comment #47
grahamcTo elaborate on the #44 issue - the regression we're seeing in Localgov Drupal after this change is because it no longer falls back to the default Views getValue() if it *is* a SearchApiResultRow but doesn't have a usable corresponding field.
I don't have the why on this part, but on this view the $real_geofield_name comes back with `location` but there's no such field in the result row. (There is a `localgov_location` field, but that returns a simple lat/long rather than a POINT(), so doesn't work here either...)
The attached patch adds the fallback behaviour back in and fixes the issue for me.
Comment #48
ekes commented> don't have the why on this part,
I can investigate further if absolutely necessary. But I think it is going to be because: (a) as mentioned, even if you index a field it is not used in the results from Search API (except in very specific Solr configurations); (b) the values that are used are from the loaded entity; (c) this does not yet include related entities, and in this case the geofield is on an entity referenced from the base entity. So you have to, as in your updated patch ask views to get the value so it can be retrieved from the relationship entity.
Comment #49
itamair commented@grahamC I am adding here a new incremental patch that applies to actual 10.2.16,
that should still retain the added Geometries support for Search API Solr and Views integration,
and eventually fallback return values coming from normal View in such uses cases as yours (that I cannot exactly reproduce on my side).
Could you test and QA that, and check if it correctly fix your reported scenario?
Comment #50
grahamc@itamair Yes! Tested on the site we originally saw the problem on, and new patch #49 resolves the issue.
Comment #51
itamair commentedOk ... patch #49 committed into 10.2.x dev branch, is going to be part of the next incoming 10.2.17 release.
Comment #52
idebr commentedOut of memory issue reported in #32 still occurs with Leaflet 10.2.20 using a Search API view. Attached patch fixes it.
Comment #53
odensc@idebr Confirmed your patch fixes the OOM issue and about doubled the performance of a page with a large map. Thanks!
@itamair Would we be able to get the patch in #52 committed? I'd be happy to make an MR for it if that helps.
Comment #54
itamair commentedThanks @idebr! Pretty cool, your patch (I QAed) is still preserving Geometries support for Search API Solr and Views integration,
and confident it could solve/mitigate also the out of memory issue ...
I committed it into the 10.2.x-dev branch, with an additional Note of mine but still setting you as Author of that,
I just deployed a new drupal leaflet 10.2.21 release with this.
Great Job! (hope it really solves the out of memory drama for the affected ones).
Comment #55
ekes commented> Out of memory issue reported in #32 still occurs with Leaflet 10.2.20 using a Search API view. Attached patch fixes it.
So far as I can see this stops loading the field data from the Index Item
Item::fieldsExtractedis basically only set onceItem::getFieldshas retrieved them. So unless something else is calling this before leaflet does, or the backend has set the data, it will be falling back to load it from the entity. This will usually work, and you won't notice unless putting break points in. It does not work for some other cases, where the field is on a referenced entity.If I'm correct that @idebr case means if the extract is being run (which happens if FALSE isn't passed, and the data has not been added by the backend already), then loading it (probably also retrieving it from the entity) within
Item::getFieldstakes more memory than directly doing agetValueon the field, which is probably a Search API issue, not one for Leaflet to solve.