Loading times decrease for directories containing 100+ of medias

Seems like you found a bottleneck. I'll investigate.
We may need to introduce pagination, like media library ..

Could you please try also with version 2.0.1 (although I guess it's the same)?
Does this happen every time, or only the first time (where the image derivates/styles are generated)?

Locally, with 1000 media entities I wait 44s for

/admin/media_directories/directory/tree/content

- if the cache is warmed-up, it takes about 25s, and with 200 media entites for 5s. Not soo tragic ;)
Anyway, seems like there might be improvement possible, fe. tree caching ..

What's your local setup? How does it perform on a server/non-dev-environment?
Do you have any other modules installed, that hook the taxonomy or media load process?

Hi,
Same problem for me, i have hundreds of images and the media browser is very slow (~20s to load, and still slow after).
It would be very useful to introduce pagination indeed !
Thanks a lot

Unsure about pagination .. we have directories/folders.

Also jsTree has no active pagination support ..

So after installing, organise your vast amount of medias into directories ;)

Well even after reorganizing our medias, it is still very slow, because each folder has hundreds of images ;-)

Hi!

I faced the same issue using the latest version of the Media directories module. When I did some debugging, I found that most slowdowns were related to the part of code that displays the items in the selected folder.

So, as I see in code, technically the files are displaying using Views item called "media_directories_base", especially it's related to building the image presets and bulk operation form items.

So, in my case, I resolved the issue by installing the module "Views infinite scroll" and adding a pager with the mode "Infinite scroll" using 45 images per page.

So, technically, after these changes, if the folder contains more than 45 media items - only the first 45 will be displayed, and the "Load more" button will be displayed in the bottom of the files list. So, a user could click on "Load more", and load next 45 items.

In my case, the performance issue was resolved, and this solution could be used but needs to be reviewed by the community as well and if it works properly and is suitable for the people - probably this dependency could be added to the codebase (I can provide a patch as well).

Funky solution to the problem sun-fire :D

I now suggest the installation of another module as workaround.
Let me think of taking this as a recommendation to the project page.

So this is now a documentation task ;)

I said task ;)

Confirm the views infinite scroll pager usage on the view works well, thanks sun-fire.

I've had this issue cause an error on two of my sites with thousands of media items and the error message is so unhelpful (http 500 error with ajax) it took me hours to realize what was happening the first time. I'd suggest adding pagination to the default view as that fixes it, though slightly less elegantly than the suggested workaround. But at least it doesn't require installation of another module or a dependency on that module from this one.

This issue seems to be coming up again and again. I have it on one site as well. Views is just so slow when trying to render a lot of entities at once. It needs some POC first to see how different it is.

So we should work on it.
I'm proposing two options:
1. Integrate views_infinite_scroll (either as a dependency or switch if the module is installed on the site)
2. Add a second listing type based on the entity query, which is way faster than Views (in both loading entities and rendering)

Thank you all for your inputs, and yes, we definitely should address this now.

Demand seems here, and we should have larger installations in mind.

Why not just introduce pagination, using the current admin themes pagination at the end of the scroll-able media area ?
I don't like the idea of having another dependency (infinite scroll, which I like a lot) - especially for something we can do our self with minimal effort.

Thinking more about that, an endless scroll would be a great fit IMO.

Why not implement our own views pagination, just having a load more button maybe having one or two settings like pagesize (bonus:auto-scroll when in viewport setting).

If we also could support the regular pager, all are still free to choose and we have the problem of directories (or ROOT) loading slow eliminated our self.

The idea of using direct entity queries would be an option for me, if loading times would increase dramatically ..
Anyway, who really needs to scroll thousands of medias, I always use the search. I can only imagine a special case, where I search for a thumbnail.

The issue is with sort I guess. We could add a pager, but because the sort direction seems to be oldest to newest, we can't see the last uploaded media which is most likely the one the user wants to select. It is an easy change.
As a user, I usually scroll down and select media there. Most of the time the media is not moved to the directory for various reasons.

I could take a quick look at how rendering with entity query could look like and give some performance differences. View performance is sometimes problematic, browsers most likely can handle rendering thousands of items just fine. Using entity query will lose some customizability some users need so we could add a setting for backend selection.

Good point with the sort, did not think about, we should definitively to change to newest first, or even expose the sort with options alphabetical/newest first/oldest first.

As I said, I would not go with an entityquery and stay with views as it offers a lot of customization just via the UI (no code).
But still, if you want, check the difference, we can introduce another setting - entityquery/regular view ;)

Whereas a pager is needed anyway IMO, as making sense, and aligns with core.

Guys, it could be funny to read "Unsure about pagination ... Locally, with 1000 media entities" in pager discussions if it isn't a problem. A view fully loads each entity, and having no pager its only a question of entities count and allowed memory when you'll get a "500: Allowed memory size ..." error.

After last module update the media_directories_base view was overridden in an update. Previously we customized the view with pager and sorts. And after the update we've got "Allowed memory size exhausted" error. We have 512M memory limit (higher than minimal recommended) and ~10 000 media entities only. It isn't much, I worked on projects where were hundreds of thousands media entities.

The pagination must be implemented. And looks like it is supported from the box - I've just added "full" pager into the view an everything works.

Yeah - this was 2021 with the mindset that we have directories (always showing everything like the file manager of your favorite OS),
and if I would have 100K of medias, I would have a ton of folders and not having all medias shown in root.
But no doubt, we will now add a pagination out of the box to reduce support in this direction.

The question remains, shall it be:

• Regular Drupal pagination
• Custom Pager
• Another module dependency

Whereas I vote for the custom pager.

And we should change sorting to newest first.

I suggest something like this:

1. Change the sort order
2. Add normal full pager (to go to specific page easily) as default
3. Show 50(?) items per page

The next question would be how to make the update - just overwriting things we need or trying to detect custom changes?

@rang501 - the view already was fully overridden recently in media_directories_ui_update_9002(). It isn't good and was a surprise for us, but if it is already done I think it could be repeated.

@dewalt Updating configuration is complicated, we have two problems here -
1. If we update it, it will override customizations.
2. If we don't update it, users must do it manually which is probably more confusing.

It is hard to find the right balance and detecting if we need to update something is not reliable. We might be able to check if some pager or sort is already added and skip the update, but for more complex changes it is just too difficult to keep track of things.

I would not introduce an update here.

This is for new installations, already running ones will have made their customization, eventually ;)

I added now the full pager (with 18 items so we get 3 rows on desktops) and the sort (by changed, ascending).

@ytsurk
I think the page size should be larger and should reflect the maximum upload size limit, otherwise, we will have a situation, where the user uploads like 20 files, and some of them are on different pages. I did test it and after selecting 20 files, the field had only 18 added.

The default PHP limit for files is 20 and I think it's something that's changed less frequently than upload size.

Good point - I really liked the fact of having three rows w/o scrolling :D

Why not make a warning on the status page, when you can upload more files then we display, with a recommendation to change the views pager size manually?

I want to check how core media handles this case. As I already committed the changes to the -dev branch we need to hurry up here ...

I would not introduce any logic, like changing the pagination size on the fly for the view.
But I could imagine, just closing our media selector, and straight return to the field inserting all uploaded medias, fe.?

Further I think, we should provide an update path now (changed my mind) - only if the view does have the no pager configured.

Core media has 24, so I suggest we take that.

An upgrade path will affect all that installed prior 2.2.2-beta4 (February 2024).
Does it make sense to do it for 2.2.0?

Closing the issue. New 3.x handles this.