Random failure: "PHP temp directory (/tmp) does not exist or is not writable to Composer."

in this issue change drupalci.yml to run `WritableFileSystemValidatorTest` many times and see if it fails randomly

💯

It did fail randomly the error i can see is while $this->inspector->getConfig('minimum-stability', $dir); and some how we get PHP as a value.

Attached a screenshot below

Now i'm wondering if the changes i pulled from 8.x-2.xis making it consistently pass, probably because of ($type === ProcessOutputCallbackInterface::OUT) the type check

This proves that with the changes from 8.x-2.x the tests are consistently passing, we can see different errors before and after merge (both cases without the check ($type === ProcessOutputCallbackInterface::OUT))

The conclusion for now,

1. I didn't see the actual error of /tmp directory for which this issue was opened for, maybe an environment problem which was fixed with a patch in the background ?
2. The tests are passing consistently now after pulling the changes from 8.x-2.x
3. Leaving this issue open for now, if the inconsistencies surface again, I also saw some more issues but they seem to be fixed by the changes pulled from 8.x-2.x
4. Unassigning and setting to active

I got this error on #3343889-17: Drop support for end-of-life versions of Composer on DrupalCI while I was not able to reproduce it on my local, re-running the test solved it for me.

Yep — all you can do for now is re-running it. That's why it's a random failure!

Per https://stackoverflow.com/a/57381161 this is perhaps happening because DrupalCI's /tmp is full? 🤔

Also: AFAICT this started happening since #3328234: Improve test DX *and* confidence: stop using VFS.

Closed the 8.x-2.x MR in favor of a new 3.0.x one. Moved all changes.

And then added more debug output.

00:02:15.868 php -r 'print sys_get_temp_dir() . "\n";'
00:02:15.945 /tmp

yet…

00:02:16.055 sys_temp_dir => no value => no value

Does that mean we can fix this by simply doing ini_set('sys_temp_dir', $some_directory_we_created)? 🤔

the problem might be with the permissions chmod 1777 the_tmp_directory might work

the open_base_dir is also empty open_basedir => no value => no value so just creating mkdir /var/www/tmp and changing permissions chmod 1777 /var/www/tmp maybe needs sudo ?

Retesting to see if it actually helped or was just a fluke

The tmp directory in not getting set

mkdir /var/www/tmp
15:15:58 chmod 1777 /var/www/tmp
15:15:58 php -r 'ini_set("sys_temp_dir", "/var/www/tmp");'
15:15:58 php -r 'print sys_get_temp_dir() . "\n";'
15:15:58 /tmp

This is now hypercritical. The failure frequency is higher. Probably because we run more composer commands now.

I guess it makes sense that calling ini_set('sys_temp_dir', …) does not help because the error message is occurring inside composer's \Composer\Console\Application::doRun():

            // Check system temp folder for usability as it can cause weird runtime issues otherwise
            Silencer::call(static function () use ($io): void {
                $tempfile = sys_get_temp_dir() . '/temp-' . md5(microtime());
                if (!(file_put_contents($tempfile, __FILE__) && (file_get_contents($tempfile) === __FILE__) && unlink($tempfile) && !file_exists($tempfile))) {
                    $io->writeError(sprintf('<error>PHP temp directory (%s) does not exist or is not writable to Composer. Set sys_temp_dir in your php.ini</error>', sys_get_temp_dir()));
                }
            });

— I bet that this is happening because we run tests with a high level of concurrency on DrupalCI, and that would make it _feasible_ that sys_get_temp_dir() . '/temp-' . md5(microtime()); runs at the exact same time in concurrent tests ⇒ a race condition!

👆 #28 means that #3345222 would help a LOT (but still would not solve it entirely): #3345222-4: Optimize Composer calls in FixtureManipulator.

🕵️‍♀️ The Composer code quoted in #28 was added in https://github.com/composer/composer/commit/28e9193e9ebde743c19f334a7294... almost 7 years ago. It was refined a few days later in https://github.com/composer/composer/commit/43eb471ec293822d377b618a4a14... and it's remained the same since then (other than non-semantical changes).

We could ask Composer to adopt hrtime() (TIL!) instead, that would make the chances of a collision far lower (microseconds vs nanoseconds).

But that will take time, which we don't have.

What if we can at runtime configure a unique temp dir? A quick experiment with a test.ini file like this:

[PHP]
sys_temp_dir = "/tmp/PID"

yields

$ php -c test.ini -r 'print sys_get_temp_dir() . "\n";'
/tmp/PID

🥳

Well I was very pleasantly surprised to discover \Drupal\package_manager\ProcessFactory — without that we might have had to make composer-stager modify its \PhpTuf\ComposerStager\Infrastructure\Service\ProcessRunner\AbstractRunner! 😅

Let's see how this fares… 🤓

So apparently tmpdir is not considered a word 🙄

While there is an ~80% failure rate today in kernel tests (in the sense that a test run is very likely to have at least one kernel test fail with the error in this issue's title), this one immediately passed … so … that suggests that #3 actually fixes this 😄

Queuing a second one to confirm… but I'm optimistic! 😄

I will land this without review from others because pretty much everyone reported being constantly blocked by this. Plus, the changes are very isolated, trivial to revert!

Yep, another all-green Kernel test suite run!

But … build & functional tests failed hard:

Exception: Warning: mkdir(): File exists
Drupal\package_manager\ProcessFactory->create()() (Line: 79)

I think that's easy to fix though 🤞

Due to the agreed hypercriticality (!) of this issue, I'm okay merging whatever fixes it post-haste without review. Consider this a "blanket RTBC".

That did it! 🥳

Both green, committed! 🚢

Now filed upstream PR with Composer to hopefully get it fixed there: https://github.com/composer/composer/pull/11375 🤞