After a fresh install of D8 and enabling simpletest, I am greeted with this error when visiting /admin/config/development/testing
UnexpectedValueException: RecursiveDirectoryIterator::__construct(/drupal/core/./modules/image/tests): failed to open dir: Too many open files in RecursiveDirectoryIterator->__construct() (line 107 of core/vendor/phpunit/php-file-iterator/File/Iterator/Factory.php).
Using OSX 10.9.2 (Mavericks) with PHP 5.4.17
After searching, I found this thread on Stack Overflow:
http://stackoverflow.com/questions/19766745/apache-php-osx-mavericks-fai...
Following the directions, I edited my /etc/launchd.conf file and added the following:
limit maxfiles 16384 32768
I then restarted the machine.
Note: you should set these values to ones that are appropriate for your machine. Setting them too high may cause problems. The default for OSX is 256. Note also that if you are setting these, you must set explicit numeric values. Setting to 'unlimited' will not take and the machine will remain at its defaults.
Not sure if this is something we can fix in Drupal but thought it wise to report in case others find a similar issue.
Comment | File | Size | Author |
---|---|---|---|
#9 | drupal8.ext-disco-rdi.patch | 2.06 KB | sun |
Comments
Comment #1
Wim LeersThis is at least major. Just like we don't want things to be broken out of the box on Windows, we also don't want that on OS X.
This is a bug that causes >256 files to be opened simultaneously. Even if that's lower than the limit on most OSes, that's still a bug.
It only happens when running tests. Possibly only when using
run_tests.sh
.I did all sorts of debugging: print stack traces, using a PHP debugger, using
lsof
, and so on. The key aspect is that theSplFileInfo
object is being passed around and seems to come with a built-in, hidden file pointer. That must be the case, because it's the only possible explanation for us reaching the max open file limit.It's being passed around as a (premature?) optimization to save time later on, to not have to create those objects later on. But my preliminary testing showed that it is not used at all! At least not for regular kernel booting, nor for running a test using
run-tests.sh
(found by adding debug output followed by anexit;
at the beginning ofExtension::__call()
).Therefore it should be safe to not pass on the
SplFileInfo
object. That actually already was safe, but if my testing was correct and representative, it should also not cause any performance regression.I also found two bugs that I had to fix as part of this solution:
getSubPath()
was being called onSplFileInfo
, but that's a non-existing method! It's a method that only exists on a the iterator.Comment #2
Wim LeersComment #3
Wim LeersComment #4
sunPHPUnit's file iterator opens plenty of directories and files that are only filtered later on, instead of within the filesystem recursion already.
→ http://www.php.net/manual/en/class.recursivedirectoryiterator.php#114504
The value is not a
SplFileInfo
object, but an instance ofRecursiveDirectoryIterator
, which provides sub-path methods.→ http://www.php.net/manual/en/recursivedirectoryiterator.getsubpath.php
SplFileInfo
is a resource that holds the file information only.ExtensionDiscovery::scanDirectory()
does open the .info.yml files viaSplFileObject
s though. Theoretically, the file handle should be closed, because the local variable is replaced with a new one, but perhaps PHP doesn't behave correctly with regard to that.Comment #5
Damien Tournoud CreditAttribution: Damien Tournoud commentedThe problem is that
RecursiveDirectoryIterator
is used in\FilesystemIterator::CURRENT_AS_SELF
, in which it returns the iterator itself when iterating.So, here:
$fileinfo
is not a standaloneSplFileInfo
but the current directory iterator itself. So, when we set it to$extension->setSplFileInfo($fileinfo);
, we are actually leaking this iterator. The directory handle for every visited directory stays open, and eventually we hit the maximum file number.(This has nothing to do with files, we are actually leaking handles to directories.)
Also, Wim is correct that
ExtensionDiscovery:scan()
is wrong in calling->getSubPath()
because this method is part ofDirectoryIterator
, notSplInfo
itself.(And
Extension
is a proxy to the underlyingSplInfo
, who designed this mess?)Comment #6
Damien Tournoud CreditAttribution: Damien Tournoud commentedCreating a standalone
SplFileInfo
should solve this nicely, but I haven't really tested this.Comment #7
Wim Leers#4: I tried precisely that. It's insufficient. The problem persists with only that change.
#5/#6: Thanks! :)
Comment #8
damiankloip CreditAttribution: damiankloip commentedThis fix looks good, and correct.
Comment #9
sunOK, #5 makes sense to me. Sorry, I was not aware that
RecursiveDirectoryIterator
keeps the directory handle open after callingaccept()
andgetChildren()
. I could be wrong, but somehow that sounds like a PHP core bug to me.In that case, we can remove the injection of
SplFileInfo
entirely.Extension
only proxies toSplFileInfo
if an actual file info method is called; e.g.,getMTime()
. This happens in e.g._system_rebuild_module_data()
and in other places.The only info that is needed for sorting purposes is the subpath from which the extension originated. For now, we can simply add that as a property like the existing
$origin
in the loop — we can clean those up in a separate issue. I already wasn't really happy with tucking the$origin
property onto the object, but ran out of energy in the original issue. Cleaning up those internals would only delay this bug fix, so let's do that in a separate issue.Comment #10
Wim Leers#9 fixes the problem. I'd RTBC, but unfortunately this wouldn't pass the docs gate:
There is no comment for
$extension->subpath
, nor isExtension::subpath
documented inExtension
.Comment #11
sunThe existing comment applies to both.
Created #2222071: Clean up addition of $origin + $subpath properties in ExtensionDiscovery
Comment #12
benjy CreditAttribution: benjy commentedThis does indeed fix the issue (which has been driving me crazy)
Since we have a follow up for the things mentioned in #10 this is RTBC
Comment #13
webchickGreat work sleuthing on this.
Committed and pushed to 8.x. Thanks!
Comment #14
Wim LeersYAY!
Comment #16
benjy CreditAttribution: benjy commentedThis issue is back for me when running tests locally. Not sure if it's related but I recently updated to Yosemite.
PHP 5.5.18
Apache 2.4.9
Comment #17
jhedstromI've started to see this popping up again as well. Temporarily disabling xdebug seems to resolve the issue.
Comment #18
benjy CreditAttribution: benjy commented@jhedstrom, i'm not sure this is run-tests fault entirely, it just adds to the problem.
Since upgrading to Yosemite i've had the issue back, I think this is the right solution: http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Mac-OS-X but i've not tried it yet.
Also note, restarting apache fixes the issue for me when it happens.