I wanted to create a new index and add it to an existing SAPI server for Solr, when I was greeted with this message:

The concept of storing multiple "virtual" Search API indexes in one Solr index (aka core) is bad practice and randomly breaks a lot of advanced features like spell checking, suggestions, automplete and others. Create a second core within your Solr server and assign this "index" to that core.

I could not save the form.

What is the reason now that we need to have a 1:1 core and index setup in Drupal 8? I can imagine on some hosts this will not be possible to do.

For instance, what if I want to create a content, media, and user based solr search? I have to create 3 separate server cores for that?

Comments

kevinquillen created an issue. See original summary.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
CreditAttribution: drunken monkey as a volunteer commented
Project: Search API » Search API Solr
Component: General code » Code
Berdir’s picture

Berdir
Primary language German
Location Switzerland
CreditAttribution: Berdir at MD Systems GmbH commented

I'm also unhappy about that. Displaying information about that and also explaining that some things might not work well, like autocomplete and suggestions and so on is one thing, but all those requirements and form validation errors are making many things really complicated.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

I noticed you can create it outside of the server, and then switch it back later under the options, but its confusing and cumbersome.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

It also creates an ERROR in the status report that won't go away.

"SOLR SERVER INDEXES
1 server
One Solr server contains more than one enabled index."

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
CreditAttribution: mkalkbrenner at bio.logis Genetic Information Management GmbH for OSCE: Organization for Security and Co-operation in Europe commented

For instance, what if I want to create a content, media, and user based solr search? I have to create 3 separate server cores for that?

If you have a closer look at Solr, you'll notice that Solr is designed this way (and like other search engines). One Server can host multiple indexes. Solr calls those indexes cores. And as I already pointed out multiple times, there are features per core like suggesters that work per core and can't deal with these "virtual indexes" Search API introduced.

I noticed you can create it outside of the server, and then switch it back later under the options, but its confusing and cumbersome.

We had to do this for backward compatibility to not break existing installations.

Displaying information about that and also explaining that some things might not work well, like autocomplete and suggestions and so on is one thing, but all those requirements and form validation errors are making many things really complicated.

I know. But this was the trade off for the 1.0 release. As you might noticed, there're other bug reports complaining about "wrong" suggestions.

Since Solr 5 there's a way to limit suggesters to not give results from all documents in core. Therefor I will drop support for Solr 4 in the upcoming 8.x-2.x branch and merge the multilingual extensions to have a config generator.
This will allow us to have these virtual indexes again, even if I don't like the concept personally. But then the trade-off will be, that you have to create and deploy a new Solr config and re-index every time you add an "index".
BTW I consider creating a new core as the easier task.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

Except when you create a 'server' in Drupal Search API, you have to define a core for that configuration to point to. That core points to a collection in Solr - which is different terminology than the index you create in Drupal. The index in Drupal is basically configuration for what should be tracked, and how, for the respective core (server in Drupal) it is assigned to. I don't quite understand why you would need to create servers to define extra core endpoints, to create more than 1 index on the same (actual Solr) server.

At the same time, to do that, you need to run a terminal command to create that new core in Solr, which I would think a good percentage of users would not have access to do on their respective hosting vendor.

I guess that is where I am confused.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
CreditAttribution: mkalkbrenner at bio.logis Genetic Information Management GmbH for OSCE: Organization for Security and Co-operation in Europe commented

This is the current mapping:

Search API Server => Solr Server AND a Solr Core
Search API Index => An index ID field (incompatible with features like suggesters, spell checker, terms, ...)

This would be the mapping I prefer and we currently somehow force in parts:

Search API Server => Solr Server
Search API Index => Solr Core

Unfortunately that doesn't match Search API's database driven architecture.

Therefor I suggested a more sophisticated approach:

Search API Server => Solr Server AND a Solr Core
Search API Index => Suggester Context

But this will require to drop spell checking (by replacing it by suggesters) and maybe the terms component. And it will require at least Solr 5.3 and a dynamic config generator.

I'm still not sure which way to go :-(

Another option would be to disable features like spell checking and suggesters as soon as you enable a second Search API index on a Search API server.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

That would be in line with my thinking as well, as long as there is an easy way from the UI to create the core (ex. sudo su - solr -c "/opt/solr/bin/solr create -c...) so that it is created proper with the right conf files. Then again, I am also assuming Solr is running in an environment that would allow that to be executed via PHP or PHP calling a shell script.

Perhaps I am overthinking it, but what I am getting at is:

1 Server:

  • Content index (backend for admin searching)
  • Content index 2 (front end, different config)
  • User index (member directory search)
  • Media index (member document search)
  • Content index 3 (event content type only with config)

Just an example..

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
CreditAttribution: mkalkbrenner at bio.logis Genetic Information Management GmbH for OSCE: Organization for Security and Co-operation in Europe commented

I don't plan to implement any command line execution because the setups are too different and and the solr service providers are completely different.
The only thing we might support in the future is Solr's own REST API.

The plan is to leverage the config file generator from search_api_solr_multilingual. You can download the complete config as zip via the UI or a drush command. The later could be used in your own command line scripts.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

Although now that I think about it, one caveat is if you have different cores (in Solr), you cannot search across them.

I just completed a project where I have 3 sites indexing content into the same shared core, so I could write a front end using Solarium to search content on any site.

kevinquillen’s picture

kevinquillen CreditAttribution: kevinquillen at Velir commented

Just as an update here, on a host like Acquia, you cannot add new solr cores without being an Enterprise level customer ($12k year+). This makes it impossible to have more than one index, which also heavily conflicts with the need for certain processors to run on certain indices (as it affects seemingly all of them on the same core).

andypost’s picture

andypost
he/him
Primary language Russian
CreditAttribution: andypost as a volunteer and at Skilld commented
Version: 8.x-1.0-rc4 » 8.x-1.x-dev
Related issues: +#2831842: Prepare usage of Solr's suggester component for autocomplete suggestions and re-enable multiple "indexes" per Solr Core later

As i see #2831842: Prepare usage of Solr's suggester component for autocomplete suggestions and re-enable multiple "indexes" per Solr Core later commited to 2.x branch so the only question is we need to wait for 2.0 or can backport that to 1.0?

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
CreditAttribution: mkalkbrenner at bio.logis Genetic Information Management GmbH for OSCE: Organization for Security and Co-operation in Europe commented

I will not backport it because suggesters are not supported by Solr 4 and early Solr 5 versions.

On the other hand, the 2.x branch could be considered stable already. The "alpha" is only valid for the new features. So you can already use it.
Since we use it in production already I'm also forced to provide update hooks for this "alpha" ;-)

andypost’s picture

andypost
he/him
Primary language Russian
CreditAttribution: andypost as a volunteer and at Skilld commented

@mkalkbrenner thanx a lot for good news!

Maybe it makes sense to add some lines to README pointing which features are per core?

I tested 2.x branch with official 5.5 solr docker & after reindexing suggestions mixed from indexes

But if project does not use autocomplete the #9 could live in one core without issues, so maybe better to show warning at status report page with a link to docs about which features could be screwed in single core?

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
CreditAttribution: mkalkbrenner at bio.logis Genetic Information Management GmbH for OSCE: Organization for Security and Co-operation in Europe commented
Version: 8.x-1.x-dev » 8.x-2.x-dev
Status: Active » Fixed

The new autocomplete plugins clearly state which ones are not reliable when running multiple indexes on one core.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.