Updated: Comment #N

Problem/Motivation

In several recent issues, we've come up with areas of Search module functionality that were not covered by the existing tests. I've also noticed that some of the test code we have is redundant.

Proposed resolution

a) Make a list of search functionality that should be tested, including:
- Base search.module functionality
- Node search
- User search
This needs to pay special attention to:
- permissions
- languages

b) Verify that this list of functionality is tested, and that there are not multiple tests for the same functionality. Add and remove tests as needed to make sure the list of existing tests match the list of needed tests.

c) Fix bugs identified by the new tests.

Inventory of functionality to be tested

Here is a list of search functionality that we need to verify has been tested.

Status: I think the list of functionality is pretty complete. Test coverage needs to be filled in.

Functionality Description Tested in OK or Needs work
Plugin system overall Modules can define plugins that define search pages
  • All of the node/user search tests implicitly test this.
  • SearchKeywordsConditionsTest verifies that a plugin can override the isSearchExecutable() method to allow searching without keywords set, and that GET query parameters are made available to plugins during search execution.
  • SearchPageOverrideTest verifies that a plugin can override the buildResults() method to control what the search results page looks like.
Test coverage is OK. Things to fix:
  • SearchKeywordsConditions test class needs better one-line and getInfo() documentation of what it actually does.
  • SearchPageOverrideTest does too -- it says it tests hook_search_page() which doesn't even exist any more.
  • These are part of #2183951: Fix up some search tests.

Search page config/CRUD Verify that you can use the UI to add/configure/enable/disable/delete search pages based on plugin types, choose the order, and choose a default page. Verify that if you go to /search you get to the set default search page. Also that you can click through to the other tabs, that disabled pages are not shown, and that the pages are in the correct order
  • SearchConfigSettingsFormTests::searchModuleSettingsPage tests that plugin-specific config can be edited in the UI, although it does not verify that this config can be used later for anything.
  • SearchConfigSettingsFormTests::testSearchModuleDisabling() tests that a default can be set by visiting the correct URL, and that it works from the search page and the search block, and that if this is the only enabled search type, the other tabs are not visible. Also verifies that with other tabs enabled, all are shown after directly going to the correct URL search/node. However, it doesn't disable via the UI but via $entity->disable() method in code.
  • SearchConfigSettingsFormTests::testDefaultSearchPageOrdering() tests that out of the box the order of search tabs at URL /search is correct.
  • SearchConfigSettingsFormTests::testMultipleSearchPages() tests that you can add new search pages with the UI, and that they are both displayed at /search, and that the order can be set via the UI (in that the order "sticks"). It also tests that the operations links are present and that after enable, disable, and delete, the search config screen has the right pages and operations visible, but it doesn't verify this is reflected at /search.
Needs tests to cover all the bases - now part of #2183989: Add tests for search admin
Search admin permissions Verify that users without administer search permissions cannot do the search page config/CRUD operations or other search admin operations Needs tests - now part of #2183989: Add tests for search admin
Search admin page Verify that you can find the admin page link on admin/config and admin/config/search Needs tests - now part of #2183989: Add tests for search admin
Search block Verify that a search from the block goes to the chosen default search page SearchBlockTest: verifies you go to search/node and that both positive and negative searches work; doesn't test what happens if the default has been changed. Needs tests for going to the default page - now part of #2183989: Add tests for search admin
Search block empty Verify that if you don't enter keywords you get an appropriate message SearchBlockTest verifies this. OK
Search block admin Verify that the search block is available on the Blocks admin page and can be placed SearchBlockTest: Verifies that the search block is available at admin/structure/block. Uses the drupalPlaceBlock() method to place it, and verifies the block appears on the page. OK
Indexing/querying system functionality Plugins can add text to the search index and then search with it
  • Core Node plugin uses this system, so Node searching tests implicitly test this.
  • SearchMatchTest - tests adding information by calling search_index() with the type set to various strings, including some in Japanese, and then querying it via the SearchQuery extender. Tests cases with single keywords, multiple AND keywords, OR keywords, negative (exclude) keywords, phrases, and complex combinations. Verifies that the different search "types" are separated in the index too.
OK
Indexing API -- search_reindex() function Should be able to use it to clear one item or the whole index. Needs tests - now part of #2184003: Add tests for the search_reindex() function
UI: reindex button Verify that the reindex button from the Search config admin page works SearchConfigSettingsFormTest tests this OK
Indexing API -- search_mark_for_reindex() function Verify that it updates the timestamp on an entry in the index
  • SearchCommentTest: Relies on extensive use of the search_mark_for_reindex function, and would fail if this function didn't mark a node for reindexing.
  • SearchNodeUpdateAndDeletionTest - verifies that after $node->save() API call and $node_search_plugin->updateIndex(), a modified node has been reindexed, and that after $node->delete() the node is immediately removed from the index and is unsearchable.
OK
Cron overall Verify all indexable search plugins' updateIndex() methods are called during each cron run. Some of the tests do run cron and would fail if NodeSearch was not being indexed. Some of the tests call the plugin updateIndex methods directly. OK
Keyword relevance for index Verify that for searches using the index, ordering by keyword relevance works.
  • SearchMatchTest kind of looks like it is testing keyword relevance ordering, but it doesn't seem to include cases where the keyword relevance is different between the results.
  • SearchRankingTest does test keyword relevance ordering in a valid way. It also tests that HTML tags influence keyword relevance rankings in the right way.
OK
Indexing/searching text preprocessing Verify that the same preprocessing (search_simplify() function) is used to process keywords during indexing and searching, with the following processing steps:
  • Lower-case everything
  • Invoke hook_search_preprocess() with the right language passed in
  • If the "index.overlap_cjk" setting is TRUE, make "words" by splitting CJK text into overlapping sequences of characters of length equal to the "index.minimum_word_size" config setting.
  • Removes all punctuation inside numbers.
  • Treats sequences of multiple . and/or - as word boundaries.
  • Removes ., _, and -.
  • Treats remaining punctuation as word boundaries.
  • Splits into words
  • Truncates all words to 50 characters.

Phew!

  • The basic CJK overlap functionality is tested in SearchMatchTest, which searches for several subsets of some Japanese text after indexing it.
  • Tokenizing all known ranges of CJK characters is tested in SearchTokenizerTest, which also verifies that English letter characters are not tokenized this way. Calls search_simplify() explicitly with the config set to tokenize at one character.
  • SearchNodeAccessTest::testPhraseSearchPunctuation tests indexing and searching for a word with an apostrophe. Why it's on this test class, I have no idea. Came from D7 that way.
  • SearchExcerptTest ends up testing some of the preprocessing stuff implicitly: punctuation in numbers, hyphen removal, punctuation as word boundaries, hook_search_preprocess()
  • SearchNumberMatchingTest implicitly tests the number preprocessing by doing node index and search from the UI for number/punctuation strings
  • SearchNumbersTest has some more test cases for numbers, including one over 50 characters long (again, makes nodes and tests in the UI
  • SearchPreprocessLangcodeTest - verifies that language is considered in preprocessing
  • SearchSimplifyTest - verifies that punctuation rules are followed in search_simplify(). Also verifies that non-word characters are removed, and that for word characters, the output of search_simplify() is at least as long as the input (nothing is removed) [Note that word and non-word are as defined in Unicode and every Unicode character is tested].
SearchNumberMatchingTest and SearchNumbersTest should probably be combined; otherwise this is probably pretty thorough. This is part of #2183951: Fix up some search tests now.
Indexing config -- minimum word length and CJK handling Verify they can be set from the Search Settings page and that the settings work correctly during indexing and during searching SearchConfigSettingsFormTest - tests that invalid values for word length (not an integer) cannot be saved, and that the default values allow the form to be submitted. Doesn't test that other values can be saved or that they do anything useful. Needs tests - now part of #2183989: Add tests for search admin
Indexing config -- cron throttle Verify the # of nodes per cron run setting can be set (may be on the search config page or the node page config page), and that if it is set it affects the cron run.
  • SearchMultilingualEntityTest uses API to set the throttle (not the UI), and assumes that the throttle should count each translation as a separate item, and verifies that the indexing is stopped after the correct number of translations is reached.
    Needs tests to use the UI for the throttle. This is now part of #2178643: Indexing status can show a negative percentage of content as having been indexed
    API function search_excerpt() Verify that an excerpt with keywords/phrases highlighted is created:
    • Only "positive" keywords should be highlighted (if you are excluding words from an advanced search, those shouldn't be highlighted even if they occur)
    • In the case that the match occurs because a preprocessed keyword matches the preprocessed text, highlighting should also work.
    • The excerpt length should be at least 256 characters and should have about 60 characters around keywords (this may be difficult to test because of the "about" part)
    • If the text is broken up, there should be ... between chunks
    • If the text is broken up, the order should match the input text order, not the keyword order.
    • All matched keywords should be highlighted
    • HTML tags in the input text should be removed
    SearchExcerptTest runs a number of test cases, verifying that HTML tags are removed and HTML entities work, that simple keywords are highlighted at the beginning, middle, and end of the text, that it works with multi-byte characters, and that it works with preprocessing-only matches. Probably OK
    Node search permissions Verify that users with/without search content/access content permissions can/cannot search nodes using any configured NodeSearch pages via the UI. Verify that users with/without advanced search permissions cannot use advanced search via the UI or via directly using advanced GET query params. Verify that node access is respected.
    • SearchNodeAccessTest looks like it should be testing node access, but all it actually does is have a module turned on that includes node access, without actually exercising any of the node access stuff. ?!?
    Needs tests #2183971: Add tests for Node search and advanced search is for adding access tests. Also, I think the method on SearchNodeAccessTest::testPhraseSearchPunctuation() should be moved elsewhere and we should either then delete this class or make it actually test node access? (see #2183951: Fix up some search tests for that)
    Node search functionality Verify that keyword searching in nodes works via the UI, and that a non-matching query gives an appropriate message.
    • SearchAdvancedSearchFormTest - verifies that node title can be searched for via submitting with the title in the keyword portion of the URL (matching and non-matching cases are tested).
    • SearchExactTest - verifies that keyword search and phrase search work as expected -- that searching for a phrase finds only the exact phrase, not any node with the keywords. It also tests that the right number of pages of results are found for a multi-page result.
    • SearchPageTextTest - verifies some error/warning messages
    OK
    Node search Advanced functionality Verify that the advanced search functionality works correctly via the UI. Should test: positive "or" keywords, phrases, negative keywords, node type filter, language filter
    • SearchAdvancedSearchFormTest - verifies that the node type filter works (find node if it matches, don't find node if it doesn't). Also verifies that you can put the title in the "Or" keywords field and the node will be found
    • SearchLanguagetest - verifies that language selections in Advanced form are added to the search URL GET params, and that configuring or un-configuring languages makes the choices available or not available. Doesn't actually test search filtering with languages though. Hence #2161067: Advanced search with Language filter does not work
    • SearchMultilingualEntityTest - calling methods on the NodeSearch plugin directly, verifies that search results can be filtered by language.
    Needs tests for the other Advanced filters.#2161067: Advanced search with Language filter does not work takes care of language filtering. #2183971: Add tests for Node search and advanced search is for the other filters that need testing.
    Node search international I'm actually not sure what the correct behavior is supposed to be here? Need to figure out what was intended
    • SearchMultilingualEntityTest - verifies that in a generic Node search you can get results in any language, and you will get a separate result for each language item that matches. This is calling the NodeSearch::setSearch() and execute() methods directly, not the UI though. When these nodes are rendered as search results you will find that they are rendered in English and not the language whose keywords matched. The output is described on #2161067: Advanced search with Language filter does not work.
    Needs definition and test I think we can leave this to #2161067: Advanced search with Language filter does not work
    Node search with comments Not sure what should be listed here?
    • SearchCommentCountToggleTest: Verifies that with various comment status settings on a node, the search results display the comment count appropriately.
    • SearchCommentTest: Verifies that comment subject and body can be searched, and that HTML tags are not rendered in the search results. Also verifies that if comments are hidden, comment body cannot be searched. Also verifies comment-related permissions with searching fairly extensively (anonymous user needs "access comments" or comments are not indexed, and searching user needs "access comments" or comment information is not shown on the search results page). Also verifies that the "Add new comment" text does not get into the search index or appear on the search page, but only for the case where anonymous users cannot comment.
    Should add a test for "Add new comment" text not getting into the index or results for the case where anonymous users can make comments. This test may fail: see #1113832: [PP-1] Comment module renders "reply" and other links in search index/results
    Node search results ordering The UI can be used to set rankings and they correctly influence search results SearchRankingTest tests that ranking by sticky, promoted to home page, keyword relevance, most recent, most comments, and most views works. They are each tested as the sole source of rankings, and there is also one combined ranking test. However, due to #893302: Search ranking based on the factor "number of comments" & "No of Page Views" is broken the tests for comments and views are currently omitted from testing. The rankings are set in the UI; search is done by calling methods on the NodeSearch plugin. OK (once related issue is committed)
    User search permissions Verify that users with/without search content/access user profiles permissions can/cannot search users via substrings on user account names, via any configured UserSearch pages via the UI. Verify that users with/without administer users permissions can/cannot search users via email address substrings. UserSearchTest (in User module lib) tests that users with/without admininister users permission can/cannot search by email, and that users with/without admin permissions can/cannot see blocked users in search results Needs more tests Now part of #2183963: Add tests for User search
    User search name functionality Verify that keyword searching for user account names works via the UI, and that a non-matching query gives an appropriate message.
    • SearchPageTextTest - verifies some error/warning messages. Note however that the dreaded "bike shed" text is not even appropriate for User Search but it is still displayed.
    • UserSearchTest (in User module lib) tests searching by user name works
    Needs test Now part of #2183963: Add tests for User search
    User search email functionality Verify that keyword searching for user account email addresses works via the UI, and that a non-matching query gives an appropriate message. Needs test Now part of #2183963: Add tests for User search
    Misc. Bug reversion tests
    • SearchEmbedFormTest verifies that you can display a form embedded in search results (via hook_preprocess_search_result() in the theme system) and the form will work (such as an Add to Cart form for Commerce).
    • SearchSetLocaleTest verifies that a locale with commas as decimal places can be set and search will still work.
    OK

    Remaining tasks

    Finish the audit. Update the tests.

    User interface changes

    None.

    API changes

    None.

    Comments

    jhodgdon’s picture

    Issue summary: View changes

    Added a couple of notes to the task list

    jhodgdon’s picture

    Issue summary: View changes

    Working on a list of functionality that needs testing (see summary).

    jhodgdon’s picture

    Assigned: Unassigned » jhodgdon
    Issue summary: View changes

    Updated the list. I think the list of functionality is probably complete. Next step is to inventory the tests.

    jhodgdon’s picture

    Issue summary: View changes

    Added some to the table. More to do but I don't want to lose this.

    jhodgdon’s picture

    Issue summary: View changes

    More additions... I'm going through the tests and have tests with names starting with SearchL* through SearchT* left to inventory, plus SearchExcerptTest which I skipped. Probably all I'll get to this week.

    jhodgdon’s picture

    Issue summary: View changes

    Found a little more time. Only 4 more to go through to see what they're testing:
    SearchRankingTest
    SearchSetLocaleTest
    SearchSimplifyTest
    SearchTokenizerTest

    jhodgdon’s picture

    Issue summary: View changes

    Finished making an inventory of existing tests, and figured out where we need more testing to be done.

    jhodgdon’s picture

    Title: Inventory search.module tests. Remove reduntant tests and add missing tests. » [meta] Inventory search.module tests. Remove reduntant tests and add missing tests.
    Issue summary: View changes

    Added some child issues.

    jhodgdon’s picture

    Title: [meta] Inventory search.module tests. Remove reduntant tests and add missing tests. » [meta] Inventory search.module tests. File issues to address test holes.
    Issue summary: View changes
    Status: Active » Fixed

    Added some more child issues. I think this inventory has been done and the required issues have been filed now. I am going to go ahead and mark this meta-issue as "fixed".

    jhodgdon’s picture

    Issue summary: View changes

    Doh! I realized that the User module had a search tests under its lib directory (the ones I had inventoried were all in the Search module lib directory). Also checked the Node lib but I didn't see anything there.

    Status: Fixed » Closed (fixed)

    Automatically closed - issue fixed for 2 weeks with no activity.