Document hard limit behavior for equal results

I get some fails on my vagrant box:

$ php core/scripts/run-tests.sh  --repeat 4 --color --class "Drupal\facets\Tests\IntegrationTest::testHardLimit"

Drupal test run
---------------

Tests to be run:
  - Drupal\facets\Tests\IntegrationTest::testHardLimit

Test run started:
  Wednesday, December 14, 2016 - 00:18

Test summary
------------

Drupal\facets\Tests\IntegrationTest::testHardLimit            45 passes
Drupal\facets\Tests\IntegrationTest::testHardLimit            44 passes   1 fails
- Found database prefix 'test34573080' for test ID 5.
Drupal\facets\Tests\IntegrationTest::testHardLimit            45 passes
Drupal\facets\Tests\IntegrationTest::testHardLimit            44 passes   1 fails
- Found database prefix 'test32834027' for test ID 7.

I try to find out more now.

My assumption is that we get these fails by a kind of random sorted facets. As you can see in the screenshot, we have 3 facets but it's strawberry (2) and not apple (2).

Yes, sorting excplitly is an option. A different approach could be to make sure that apple has a different amount of results in facets than strawberry. Currently they both have 2 results.

With the patch applied i still get these random test fails on my local machine. 2 out of 8 tests are failing. It seems like my assumption regarding sort was not correct. I would like to debug that issue on my local machine to find the root cause. Should we reopen that issue?

I just want to give an update and say that i am still working on it. One thing that is blocking me, is that i cannot enable hard limit due to this check:

    if (strpos($facet->getFacetSourceId(), 'search_api') === FALSE) {
      $form['facet_settings']['hard_limit']['#disabled'] = TRUE;
      $form['facet_settings']['hard_limit']['#description'] .= '<br />';
      $form['facet_settings']['hard_limit']['#description'] .= $this->t('This setting only works with Search API based facets.');
    }

Maybe i am doing it wrong but my facets source is views_page:testindex__page_1. Is my problem related to #2772745: Search API integration doesn't check/define feature support of backends ?

What is weird is, that the sort order of facets is correct without the hard_limit:

And is incorrect with the hard limit set:

One assumption (no proof for it) might be, that the limiting is done before sorting? I would like to dive in deeper and debug the actual hard limit (and not the test). If i could solve the problem about (facet source id check) i could check that.

I could reproduce it, but i still do not know the root cause. :-)

I created the following facets scenario:

• Atom (2)
• Bar (3)
• Beer (2)
• Clown (3)

i disabled cache in Drupal\views\ViewExecutable::execute temorarly :

   if (FALSE && $cache->cacheGet('results')) {
      if ($this->pager->usePager()) {
        $this->pager->total_items = $this->total_rows;
        $this->pager->updatePageInfo();
      }
    }
    else {
      $this->query->execute($this);
      ....

After that i reloaded my search page several times and got this result:

https://www.drupal.org/files/issues/random-facets.mp4

For any reason \Drupal\search_api_db\Plugin\search_api\backend\Database::getFacets() returns these facets randomly with hard limit enabled on my machine.

I could narrow it down to the sql query that generates the facets. In my case its this query, generated and called in \Drupal\search_api_db\Plugin\search_api\backend\Database::getFacets:

SELECT t_2.value AS value, COUNT(DISTINCT t.item_id) AS num
FROM
(SELECT DISTINCT t.item_id AS item_id, '1000' AS score
FROM
search_api_db_test t) t
INNER JOIN search_api_db_test_field_tags t_2 ON t.item_id = t_2.item_id
WHERE t_2.value IS NOT NULL
GROUP BY value
ORDER BY num DESC
LIMIT 3 OFFSET 0

This query gives me the facets with the same quantity in a random sort order:

[vagrant@vagrant facets]$ drush sql-query "SELECT t_2.value AS value, COUNT(DISTINCT t.item_id) AS num FROM (SELECT DISTINCT t.item_id AS item_id, '1000' AS score FROM search_api_db_test t) t INNER JOIN search_api_db_test_field_tags t_2 ON t.item_id = t_2.item_id WHERE t_2.value IS NOT NULL GROUP BY value ORDER BY num DESC LIMIT 3 OFFSET 0"
3	3
5	3
1	2
[vagrant@vagrant facets]$ drush sql-query "SELECT t_2.value AS value, COUNT(DISTINCT t.item_id) AS num FROM (SELECT DISTINCT t.item_id AS item_id, '1000' AS score FROM search_api_db_test t) t INNER JOIN search_api_db_test_field_tags t_2 ON t.item_id = t_2.item_id WHERE t_2.value IS NOT NULL GROUP BY value ORDER BY num DESC LIMIT 3 OFFSET 0"
5	3
3	3
1	2
[vagrant@vagrant facets]$ drush sql-query "SELECT t_2.value AS value, COUNT(DISTINCT t.item_id) AS num FROM (SELECT DISTINCT t.item_id AS item_id, '1000' AS score FROM search_api_db_test t) t INNER JOIN search_api_db_test_field_tags t_2 ON t.item_id = t_2.item_id WHERE t_2.value IS NOT NULL GROUP BY value ORDER BY num DESC LIMIT 3 OFFSET 0"
3	3
5	3
1	2
[vagrant@vagrant facets]$ drush sql-query "SELECT t_2.value AS value, COUNT(DISTINCT t.item_id) AS num FROM (SELECT DISTINCT t.item_id AS item_id, '1000' AS score FROM search_api_db_test t) t INNER JOIN search_api_db_test_field_tags t_2 ON t.item_id = t_2.item_id WHERE t_2.value IS NOT NULL GROUP BY value ORDER BY num DESC LIMIT 3 OFFSET 0"
5	3
3	3
1	2

To find a solution for that problem, we must decide how the hard limit should behave in such cases. If there is a hard limit of 3 but there are 4 facets with the same quantity of results, which one should be cut away? The sort options defined in the facets settings do not help here, as the processing (sorting) applies much later. E.g. we do not have the translated entities label here. I am not a facets pro but i see the following solutions for that problem:

1. Sort by the value as the second sort dimension in the query above (maybe only if a limit is set). E.g. ORDER BY num DESC, value ASC. This would prevent randomness, but might be a non transparent to the user, as the selection bases on ids. But still better than random sort.
2. Perform the limit later in the process - maybe after all processors (sort) have run.
3. Leave it as it is and inform the user that there might be edge cases on db backends, when there are facets with the same quantity.

I would try to go for option 1 + user info, but would be happy to get feedback on this.

Thank you @borrison_ i really like to help here and try to finish things i started :) Thank you as well for always giving such a good and quick feedback.

I created #2836994: Fix random test fail in facets module in search_api with a patch to get 2nd sort dimension in. In the patch attached is a new method to assert a certain sort order of facets. That code helped me debugging this issue and could be useful for other tests. With the patch reported in #2836994: Fix random test fail in facets module i had 200 test-runs without a single fail.

There is also a text for the README.txt, which might be improved (as i am not a native speaker).

Ups, seems like my new assertions are not working with the restructured test framework, so i reverted them again. I also corrected the line-length in README.txt

This should be the right patch. Messed up my local repo a bit. Sorry for that :-)

#2836994: Fix random test fail in facets module was committed yesterday. So we could add the description in #23.