Package Manager actions intermittently blocked on shared hosting

While this might be a task, it seems at least major if package_manager can't work on one of its main target environments (assuming similar limits exists on other shared hosts). I also personally feel it needs to hard block any attempt to charge money for site templates because people actually need to able to successfully use those sites on cheap hosting - if they can't they deserve their money back.

Did some quick numbers:

There are about 20k files in Drupal's /core directory

find ./core -type f | wc -l
19855

Of these, 4.2k are test classes

find ./core -type f -name '*Test.php' | wc -l
4225

There are a further 1.2k files in system's test module directory:

 find ./core/modules/system/tests/modules -type f | wc -l
1242

Just in Umami there are over 500 files, probably a lot of this is config:

find web/core/profiles/demo_umami -type f | wc -l
576

For comparison there are more like 3k files in /vendor from all of our production dependencies put together.

find vendor -type f | wc -l
2911

So it looks to me like if we remove test modules and concrete test files from the core subtree split, we could drop at least one quarter of files for production Drupal sites, but it might be with more investigation that we could drop even more of that. We could also look into dropping tests from packages for contrib modules too.

There has already been an issue open for that since 2019, so actually trying to do it should happen there: #3067979: Exclude test files from release packages.

But there might be other things going on.

I don't know off the top of my head which flags package_manager uses for rsync, but is it setting --no-compress and is it using delta-transfer or not? (I think it defaults to off for local directories, so would have to be explicitly flagged back to on to use it).

Also, could we look at permanently leaving a sandbox in place, and then relying on rsync --delete etc. to remove any cruft in the sandbox the next time it's used?

We probably also need an issue to standardize test module location.

Most are in modulename/tests/modules

But some like options are in options/tests

I have a few clients where I just delete core/tests
and
core/*/tests

I should add
umami
core/profiles/tests

I've opened #3534109: Permanently maintain a sandbox directory in package_manager, which is going to be non-trivial to implement but I think might be the overall root cause here, although it needs to be combined with other issues to remove the total amount of files too.

Did some googling for shared hosting limits, and have a theory for what the actual problem might be.

Lots of shared hosts have an inode limit per account, e.g. this reddit thread talks about 250k inode limit. https://www.reddit.com/r/webhosting/comments/2y4qa4/a_question_about_ino...

On #3534109: Permanently maintain a sandbox directory in package_manager I'm estimating about 30,000 files in Drupal CMS, so when that's copied to the sandbox directory, that's 60,000 - this is obviously way under 250k and ought to not trip the limit.

However, I then found https://support.hostinger.com/en/articles/1583210-what-is-the-inode-limi...

And more importantly https://support.hostinger.com/en/articles/1583491-why-does-the-number-of...

So if hosting.com has a similar system, the following can be happening:

1. Every time something is done via project browser, a sandbox directory is created and deleted.

2. Creating the directory adds 30,000 files to the inode limit.

3. cpanel or whatever inode tracking is buggy, and does not always remove files from the inode limit when they're deleted.

4. This keeps happening until the hosting account thinks there are 250,000 files, even if there are still 30,000 or 60,000 - and the account gets locked.

5. You open a support ticket, and someone in hosting support goes and clicks the 'recalculate file limit' button which resets it to the actual amount of files on disk.

And this also might explain why they wouldn't tell @pameela what the problem was directly, because the actual limit was not reached, but instead the sheer number of file operations triggers a bug on the hosting platform itself, which they probably don't want to advertise.

hosting.com does say publicly they have an inode limit of 600,000 files https://kb.hosting.com/docs/inode-count

If I'm right, then it only requires twenty-ish package_manager operations to potentially hit that limit assuming they fail to register deletions consistently.

It might also possible to tell if this is the right diagnosis by breaking the account, and then checking the inode limit in cpanel, vs. counting the number of files on disk (instructions for both are on that hosting.com page). If cpanel says 600,000 and the command line says 30,000 or 60,000 then that has to be what's going on. Although it might be that what cpanel says and what hosting.com uses for the actual limit are not the same thing, so even if cpanel shows 30,000 something else might have hit 600,000.

cpanel does allow for configuring inode limits at the cpanel level, and found one bug report about it getting cached https://support.cpanel.net/hc/en-us/articles/4403787481623-cPanel-STATIS...

@pameela I don't think it's actually hitting the file limit, but it could be incrementing the inode limit without decrementing it per the links above - the actual number of files remains OK but the host thinks there's more.

But also, #3534109: Permanently maintain a sandbox directory in package_manager would dramatically reduce the number of file operations that package_manager makes in almost all circumstances, so even if it's another limit like disk operations, CPU etc. being hit, there is a good chance of it helping with that anyway.