Problem/Motivation
Our directory listing pages were taking a very long time to load - anywhere from 6 seconds on a good day to 30+ seconds when under load.
I profiled the code and tracked most of that time down to this function call in MetadataEventSubscriber::setMetadata():
$data = $this->createData($name, $fid, $file, $subdir_fid);
$query = \Drupal::entityQuery('filebrowser_metadata_entity')
->condition('fid', $fid)
->condition('module', 'filebrowser')
->condition('name', $name);
$entity_id = $query->execute();Adding an index to the table reduced the page load time to around 2.5 seconds on a good day (not sure about a bad day yet), and the total database query time from around 4 seconds to just 235 ms (measured by WebProfiler).
CREATE INDEX fid_module_name USING BTREE ON filebrowser_metadata_entity (fid, module, name);
Steps to reproduce
Not sure exactly... We have 34,033 records in that particular table, so maybe just having a lot of files?
Proposed resolution
If adding the index automatically is an option, I would recommend it.
Otherwise I guess this is just for others who might have the same problem to try.
Thanks!
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #7 | 3361551-1.patch | 724 bytes | clivesj |
Comments
Comment #2
clivesj commentedWe had a problem with bloated tables, but this was resolved a long time ago.
Maybe this is a legacy db that was not corrected yet?
I will close this one, please re-open if or when needed
Comment #3
mi-dave commentedThanks for your input.
Without knowing what the bloated tables problem was, or what would class as a legacy DB, I can't say for sure. The site is several years old for sure, but we've kept the module up-to-date and run all the database migrations.
We have since added Memcache and moved to faster hardware, so I can't do a direct comparison with the original load times - but when I remove the index now, the page load time jumps from around 1 second up to 5 seconds (consistently).
We currently have 52,473 rows in the table, of which 21,820 relate to the node I am testing on (10,661 files across 932 directories & subdirectories). We are using Filebrowser 3.1.4, Drupal 10.4.7 and MariaDB 10.6.21 (Ubuntu 22.04).
I've re-opened the issue as you suggested - but if it's not affecting anyone else and isn't worth investigating further, I'm happy for it to be closed, as I have a solution that's been working well for us for the last 2 years.
Comment #4
clivesj commented1 to 5 seconds is significant.
I will put issue on my personal list for further investigation.
Did you solve it within Filebrowser by a patch or did you do it otherwise?
Comment #5
mi-dave commentedThanks!
I manually added the index to the MySQL table using the SQL above.
Comment #7
clivesj commentedI ran a test on a table with 100.000 rows and the performance gain is indeed enormous.
It was an easy fix so I committed it to todays dev and also provide a patch here
Comment #8
clivesj commentedComment #9
vtrefny commentedHi, the database update fails for me after upgrading to 3.1.5 which seems to be related to this fix:
Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near "USING" LINE 1: CREATE INDEX fid_module_name USING BTREE ON filebrowser_meta... ^: CREATE INDEX fid_module_name USING BTREE ON filebrowser_metadata_entity (fid, module, name); Array ( ) in filebrowser_update_9106()Drupal 10.4.7, PHP 8.2.28 and PostgreSQL 15.13