Currently the Search API stores a lot of unnecessary stuff in the options settings (making #1228726 necessary). This makes it painful when working with Features and introducing some small changes to the index that then might lead to large differences in the export code.
Ideally, only the fields that get indexed are stored in the options settings.

Comments

drunken monkey’s picture

drunken monkey’s picture

Title:Reduce Search API index options setting size» Reduce size of stored index fields settings
Category:bug» task

(Clarification: We won't touch options besides the fields settings here, I think.)

Current plan:
- Remove "name" and "indexed" key from field settings.
- Only store indexed fields in settings.
- Provide an API method (SearchApiIndex::getFields()) to access the previously available information (with unindexed fields, names and additionally descriptions).
- Related fields will be added via a special $index->options['additional fields'] entry. (This also means unused related entities will now automatically vanish again when the fields form is saved.)

Patch is under construction, it's not too trivial.

drunken monkey’s picture

- Store "boost" only for fulltext fields, and if it is different from "1.0".

drunken monkey’s picture

Title:Reduce size of stored index fields settings» Reduce size of stored index settings

(Clarification: We won't touch options besides the fields settings here, I think.)

Disregard that – just checked, and for me about 2/3 of memory size (after the above changes to the fields settings) come from the „Aggregated fields“ data alteration alone. While, for testing purposes, I do have several aggregated fields defined, this is of course a tremendous waste of space.
So I'm gonna apply the above general rules also to those settings. This shouldn't have any impact on other modules, however, as I guess none of them manually checks the internal settings of that data alteration.

drunken monkey’s picture

Status:Active» Needs review
StatusFileSize
new41.52 KB

Most processors also have a "fields" setting, this was also shrunk down to the essentials.

The attached patch works for me (with patches to several other modules, which I'll post in a minute to those) and reduces the stored size of the index options by a factor of about 10 (again, for me).

drunken monkey’s picture

mh86’s picture

Status:Needs review» Needs work

Tested your patch (in combination with the Solr Search) and the Features export code for Search API index fields was reduced from around 1000 to 25 lines, yeah :)

One small problem I've found so far:

Notice: Undefined index: name in theme_search_api_index() (Zeile 761 von search_api/search_api.admin.inc).

--> the indexed field names in the search index view page are missing.

drunken monkey’s picture

Status:Needs work» Needs review
StatusFileSize
new43.07 KB

Ah, you're right, missed that one.

drunken monkey’s picture

Status:Needs review» Fixed

Committed.

dimarick’s picture

StatusFileSize
new28.61 KB

I am get a problem with this commit, becouse field names are deleted from index config, but search_api_views is not fixed.
See attach for details

dimarick’s picture

Status:Fixed» Needs work
Damien Tournoud’s picture

I had a lot of issues with that too. It seems that this commit partly removed the fields settings from the index configuration (?!?).

maxmic’s picture

I had a lot of issues with that too. Also with search_pages modules and the extend_searche_page.

drunken monkey’s picture

Status:Needs work» Needs review
StatusFileSize
new3.52 KB

I am get a problem with this commit, becouse field names are deleted from index config, but search_api_views is not fixed.
See attach for details

Ah, yes, sorry, missed that spot. Patch attached.
The fix is already contained in #1231512: Use real Relationships instead of level magic in Views integration, too.

I had a lot of issues with that too.

You'll have to be a bit more specific there, I fear.
However, I've already spotted two bugs myself, which should be fixed by the attached patch.

It seems that this commit partly removed the fields settings from the index configuration (?!?).

As previously listed:

- Remove "name" and "indexed" key from field settings.
- Only store indexed fields in settings.
- Provide an API method (SearchApiIndex::getFields()) to access the previously available information (with unindexed fields, names and additionally descriptions).
- Related fields will be added via a special $index->options['additional fields'] entry. (This also means unused related entities will now automatically vanish again when the fields form is saved.)
- Store "boost" only for fulltext fields, and if it is different from "1.0".

maximpodorov’s picture

Field selection (in "Processor settings" section) is lost after saving settings in index workflow page. Is it possible to fix this?

drunken monkey’s picture

StatusFileSize
new4.43 KB

Ah, yes, thanks for catching that! I wholly blame Form API's many pitfalls.

Extended patch attached.

maximpodorov’s picture

Thank you. Th problem #15 is fixed.

mh86’s picture

Status:Needs review» Needs work

Just realized that for one index some of my facet blocks are gone (I guess it's related to this issue). This only affects those from related entities (e.g. Profile -> Field collection -> Taxonomy terms), the others ones (fields on the base entity) are still there.

mh86’s picture

Furthermore, all my views field with '(indexed)' (in combination with #1231512: Use real Relationships instead of level magic in Views integration) disappeared.

drunken monkey’s picture

Just realized that for one index some of my facet blocks are gone (I guess it's related to this issue). This only affects those from related entities (e.g. Profile -> Field collection -> Taxonomy terms), the others ones (fields on the base entity) are still there.

Define „gone“.
If they still appear on the „Facets“ tab, but (when enabled) nowhere on the „Blocks“ page, this also happens to me from time to time and seems to be a weird bug in the Facet API. Something that should never be NULL is suddenly set to NULL for some facet blocks and chaos ensues.

However, I can't reproduce either of these two, so I'll need a bit more details. Let's just discuss this this afternoon.

pbuyle’s picture

StatusFileSize
new4.83 KB

I had several "Undefined index: value" and "Undefined index: original_value" error preventing correct indexing when using an Features exported index in SimpleTest that I traced to usage of $this->options['fields'] in SearchApiIndex::index. Replacing it with $this->getFields() fixed the issue.

Here is an updated version of the patch in #16 with this additional fix.

drunken monkey’s picture

I had several "Undefined index: value" and "Undefined index: original_value" error preventing correct indexing when using an Features exported index in SimpleTest that I traced to usage of $this->options['fields'] in SearchApiIndex::index. Replacing it with $this->getFields() fixed the issue.

If you executed update.php properly, this shouldn't happen. Using $this->options['fields'] should be completely correct in this case, as we only need the indexed fields' 'type' keys.
How exactly did you trace this? And what do you mean with „using an Features exported index in SimpleTest” – did you try to import an index that was exported before this change? This of course won't work, like with all changes that require DB updates.

Also, I'm pretty sure you mean original_type, not original_value.

drunken monkey’s picture

Status:Needs work» Needs review
StatusFileSize
new6.5 KB

Spotted the bug in getFields() that caused the whole trouble for Matthias. And once again it was the type of bug where you wonder afterwards that anything at all worked … Quite a miracle all worked for me – maybe I should rethink my testing setup.

Anyways, patch with the fix attached, please test!

mh86’s picture

Status:Needs review» Reviewed & tested by the community

Thanks Thomas for taking a look at it. My views fields and facets are back now. And if you need some complex configs for testing, let me know ;)

drunken monkey’s picture

Status:Reviewed & tested by the community» Fixed

Great, committed.
Hopefully that's all fixed now.

drunken monkey’s picture

Status:Fixed» Needs review
StatusFileSize
new748 bytes

No, not yet. :-/

See #1111852-105: Add a "More like this" feature – the „More like this“ config form has still some wrong usage. Patch attached.

drunken monkey’s picture

Neither API change nor release blocker now, though.

modstore’s picture

Fixed the problem for me. Cheers.

drunken monkey’s picture

Status:Needs review» Fixed

Great, committed.
Thanks for testing!

BeaPower’s picture

Is this in any available releases such as stable or dev - or not yet?

drunken monkey’s picture

It's in RC 1 and dev.

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.