At Drupal Europe, I (Joris) guided a core conversation around getting Search API into Drupal Core. There were 2 main drivers for this.
The first driver behind this question were the factual numbers. Today, Search API has 30k installs for Drupal 8 and 70K installs for Drupal 7. Also, Search API is mainly maintained by 5 contributors, split between Search API, Search API Solr & Facets and some others. This bus factor is obviously high and is one of the concerns we would like to address.

Next to that, the current core search module has a few open issues with feature requests of things that already in Search API. There are loads of agencies that are using Search API to provide them with a good base for creating awesome search experiences. To decrease the community split between the 2 modules and to enable a more powerful solution out of the box, we could leverage some of the Search API code

There are different approaches we can take to resolve this problem. We can a do a very simple copy/paste of the module into the core directory, deprecate the current search and upload a patch. We don’t think that’s very likely to be given a green light, and doesn’t provide a transition path.

Another option to resolve that community split is to deprecate the current core search and point people wanting to create searches towards Search API. This is probably the easiest to do on a technical level. It would be a first for drupal core to point towards a contrib module (or contrib module ecosystem) as a solution for something. This unfortunately doesn’t solve the bus factor issue.

We could also copy the API bits of Search API into core and leverage them from Search API’s perspective, disregarding Drupal Core Search. This is bad, because it leaves the core search in the same state and only helps for the bus factor, but has no added value for anyone not using Search API.

The last solution we could think of, is to move parts of Search API that are stable, well-tested and uncontroversial to core and improve core search to work with them. We have to figure out which parts of Search API that is and what internal dependencies they have. When we do that, we can make current search better and decrease the surface of Search API. Eventually this hopefully means that we can keep the UI in contrib-land during the drupal 8 cycle, perhaps even for Drupal 9.
This is the solution that we (maintainers of Search API ecosystem) think is the path of least resistance towards improving the current state and benefits both problem-spaces that we are trying to resolve.

Usecases we are targeting to improve:
  1. Improve the current core search code by letting it use classes and components that are carried by more hands.
  2. Ensure the stability on the Search API ecosystem by handing parts off to core.

Usecases we are not targeting to improve:

  • Improving the current search UX/HTML output
Sign-offs given:
None yet.
Meet the initiative team
  • Joris: Doing the things (Facets maintainer)
  • Thomas: Doing the rest of the things (Search API maintainer)
  • Jimmy: Help with doing the things (Facets maintainer)
  • Nick: Coordinating the things (Search API maintainer)
  • Markus: Test the things with other backends (Search API Solr maintainer)
What exactly are we planning to do?

Parts that we propose can move from Search API to core:
- Tracking
- Indexing

These are parts of Search that can also be used by core and they are the most stable parts of Search API. They are small(er) and probably the least controversial bits. That doesn’t mean we don’t want the other components in there, but this makes this conversion lighter and easier to grok.

How we are working

Not yet, but you can find us in #search on drupal slack. That’s where the Search API team hangs out most of the time. Some of us are in the rocket chat (drupalchat.me#search)

As soon as this is approved, we will create issues for the parts we want to solve in the core issue queue.

How you can help

  • Help define/confirm the path forward.
  • Help write documentation/code for the new features.
  • Help us understand the process on pushing this forward.

Comments

borisson_ created an issue. See original summary.

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented

Confirming I'm supporting this initiative.

mkalkbrenner’s picture

mkalkbrenner
Primary language German
Location 🇩🇪
commented

me too

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

I’m also on board with this – and curious to hear people’s opinion on it.

sam152’s picture

sam152 commented

Wow, really interesting proposal, thanks for taking the time to write up this summary.

The last solution we could think of, is to move parts of Search API that are stable, well-tested and uncontroversial to core and improve core search to work with them.

Can the whole initiative be framed as "make search API irrelevant by improving the feature set of the core search module"? Why limit the scope to APIs and leave the UI off the table for contrib to support?

In reality, with such an approach all of the features and components that make search API the powerful and ubiquitous tool it is today would slowly be ported into the core module as (possibly individual and isolated?) enhancements, leaving the existing functionality and APIs intact and deprecated where appropriate. I wonder if the whole initiative would be more tangible if that could be broadly road-mapped, like a series of large overarching meta issues that demonstrated how we could get from A to B.

A back of the napkin example (which probably trivialises the domain) might be something like:

  1. Allow search results to be segregated by user defined indexes.
  2. Allow search module to index individual entity fields.
  3. Allow individual fields indexed by search to be processed before indexing.
  4. Allow users to select and replace the search back-end used for each index.
  5. ...

I think the main question to answer with that approach is: could these features properly stabilise in minor releases or would it be best served using the experimental module approach. It would be amazing if this could end up occupying the "search" module namespace in the end and we somehow had a plan for full BC with the current feature-set and APIs. I suppose it rests on how much these things could be architected in an additive fashion against the current feature set.

Just my 2c! Thanks again for all the work that has gone into this ecosystem.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

Can the whole initiative be framed as "make search API irrelevant by improving the feature set of the core search module"? Why limit the scope to APIs and leave the UI off the table for contrib to support?

Yes, that sounds like a good summary. But like we mentioned in the IS, this is only one of the possible routes we see moving forward. I don't want to assume that this is the best path towards success.

I think the main question to answer with that approach is: could these features properly stabilise in minor releases or would it be best served using the experimental module approach.

I personally think that we don't have go trough the full experimental module hoops, currently search api in contrib is quite stable. We don't see a ton of structural bugs appearing, a big chunk of our recent commmited issues are with integrations of other modules (views mainly) and adding new features.

It would be amazing if this could end up occupying the "search" module namespace in the end and we somehow had a plan for full BC with the current feature-set and APIs. I suppose it rests on how much these things could be architected in an additive fashion against the current feature set.

For me, this is also a big open question. This is also the reason why one of the approaches we suggested is just a deprecation of current search.

k4v’s picture

k4v commented

This is a brilliant idea :).

joel_osc’s picture

joel_osc commented

I am 100% behind this!

izus’s picture

izus commented

I also think search_api have proven that its place is in core :)
Have the search_api module as a core experimental module + Deprecate core search + point to search_api (that would be a core experimental module) seems to me a good first step

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

Core's search module has not received significant improvements in a long time. I know for a fact that it was struggling to keep up with API conversions during the D8 development cycle. I'm not a search expert myself, but I remember blog posts pointing out how core Search (search) is not meeting expectations in various ways.

I do think it's important that Drupal can offer search out of the box, without requiring additional software like Apache Solr. But if core Search would be based on the same foundation, I think that'd be a big benefit. And of course, Search API already includes the "Database Search" submodule which does exactly this.

WRT today's core search: to what extent could that be migrated automatically to search_api + search_api_db today? If search_api_db would not only offer a superset of what search does, but also offers an automatic update path to it (basically: import core config), then this becomes a lot more feasible. I do not think we must continue to support core search's exact configuration model.

borisson_’s picture

borisson_
Primary language Dutch
Location Mechelen, 🇧🇪
commented

If search_api_db would not only offer a superset of what search does, but also offers an automatic update path to it (basically: import core config), then this becomes a lot more feasible

This would be possible in terms of features. Recreating the same UX as core search does with search api would be a struggle or would require custom code to build, the filters provided by core search (especially that provided by the user search) would require custom forms to maintain the same html structure.

wim leers’s picture

wim leers
Location Ghent 🇧🇪🇪🇺
commented

to maintain the same html structure.

I don't think this is a requirement (although that surely makes it simpler from a release management POV). What is a hard requirement I think is that functionality remains unchanged. If the HTML looks different, that can be acceptable, because during the D8 cycle search would at most be deprecated, and only in D9 would search be removable. Hence we're giving site owners a very long period of time to move from one to the other.

andypost’s picture

andypost
he/him
Primary language Russian
commented
Related issues: +#3075695: Move search_index() and related functions to a service
damienmckenna’s picture

damienmckenna
Location TN, USA
commented

Some of the same "advanced search" fields could be replicated with Views exposed filters, right?

Personally I'd like to see Facets added too, but that could be a "phase 2" milestone.

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented

Some of the same "advanced search" fields could be replicated with Views exposed filters, right?

Yes, that would all be possible with exposed filters. (However, the Keywords section might not work exactly as expected/desired when you enter terms in more than one input field.)
It would just need some custom theming to have the “Advanced search” box collapsible, and with columns, as it is currently.

catch’s picture

catch
he/him
Primary language English
commented

I think this needs to be revisited in terms of starshot.

If starshot includes search_api (not a given, but it's likely to be discussed), then the 'out of the box' Drupal experience for new users won't include core search module. A big reason for search being in core is precisely that out of the box experience.

Assuming that starshot does include search_api, this leaves three possibilities:

1. search_api in starshot, search module in core (and not enabled in starshot). IMO this would be very bad.

2. search_api starts moving into core, starshot includes search_api + maybe extra contrib modules. (basically what this issue proposes)

3. search_api in starshot, search module moves directly to contrib, no search in core (would break help topics search integration, maybe other things).

andypost’s picture

andypost
he/him
Primary language Russian
commented

Moreover the help module's topics using core search implementing HelpSearch plugin and have tricks with cron-indexing

quietone’s picture

quietone commented
Project: Drupal core ideas » Drupal core
Version: » 11.x-dev
Component: Idea » other

The Ideas project is being deprecated. This issue is moved to the Drupal project. Check that the selected component is correct. Also, add the relevant tags, especially any 'needs manager review' tags.

catch’s picture

catch
he/him
Primary language English
commented
Related issues: +#3476883: [Policy, no patch] Move Search module to contrib

Now that search_api is included with Drupal CMS I think we should look at moving search to contrib and then separately at moving search_api into core. This would require less actual backwards compatibility work because we can deprecate search module instead of all of the individual APIs.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.