Problem/Motivation
PHP 5.4 made a number of changes to its concept of stream wrappers. This lead to a new proposed stream wrapper class, see http://php.net/manual/en/class.streamwrapper.php. However, our copy of that proposed class, \Drupal\Core\StreamWrapper\PhpStreamWrapperInterface, has not been updated to PHP5.4. This leads to many warnings and (silent)) failures.
[Note: this issue started as a follow-up of #2107287: PHP 5.4 calls a new stream_metadata() method on stream wrappers not implemented by Drupal, but was turned into a meta for that issue and possible others that can be created out of this one. Work here and in the other issue can proceed independently form each other.]
The goal of this issue is to find out about:
- [done] All inconsistencies between the PHP 5.4 streamWrapper class and our interface Drupal\Core\StreamWrapper\PhpStreamWrapperInterface.
- [done] The errors that these omissions give.
- [done] Solve these bugs.
Inconsistencies between the PHP proposed streamWrapper class and our PhpStreamWrapperInterface
Methods and properties of :
- /* Properties */
-
public resource $context ;
- /* Methods */
- public resource stream_cast ( int $cast_as )
- public bool stream_metadata ( string $path , int $option , mixed $value )
- public bool stream_set_option ( int $option , int $arg1 , int $arg2 )
- public bool stream_truncate ( int $new_size )
Note: from the parent issue (comment #3) we know that there are classes (that implement PhpStreamWrapperInterface) that implement one or more of the above missing methods anyway, but not in a satisfactory way. E.g. LocalStream returns false for stream_cast(): against the proposed return type and a silent failure.
Errors and warnings due to these omissions and silent failures
In general:
- Not implementing a method listed in the proposed stream wrapper class will lead to warnings like "Warning: chmod(): DrupalPublicStreamWrapper::stream_metadata is not implemented!"
- Not implementing a method listed in the proposed stream wrapper class will subsequently result in a (silent) failure to apply the requested operation.
Per method:
- stream_cast: curl_setopt() will fail, but is just an example. All functions that need to work on the real file handle will fail. Those are these functions that probably need to pass it to the OS or another extension not working with PHP's stream wrapper concept.
- stream_metadata: See parent issue for more errors due to missing stream_metadata().
- stream_set_option: this is called in advanced use cases where extremely fine grained control over stream access is needed.
- stream_truncate: chown() and chgrp() on streams will fail, as did chmod() before the child issue got committed to D8.
- stream_truncate(): ftruncate() on a stream will fail.
- stream_set_option(): stream_set_blocking(), stream_set_timeout(), and stream_set_write_buffer() will fail.
Proposed resolution
- [done] Provide tests that show the failures in PHP5.4
- [done] Define the full interface in our StreamWrapperInterface.
- [done] Implement the whole interface with meaningful and correct implementations, and without silent failures, throughout the whole inheritance chain of Drupal stream wrappers.
Note: stream_set_option() could not be made to work on Windows. Therefore we opted for an explicit failure with a clear error message (following the stream wrapper implementation guide lines). As it only implements advanced file usage, missing this is not a real problem.
Remaining tasks
- Review.
- Backport to D7.
User interface changes
None.
API changes
The actual API changes are minimal and exists of additions only: addition of some methods and addition of some optional parameters.
However, stream wrapper methods are normally not called directly, but via the standard available PHP file functions (fread, ftruncate, chmod, etc.). So the importance of this issue is not in the API changes but in correcting a broken file handling system. It is not us that came up with these changes, but PHP 5.4.
Beta phase evaluation
| Issue category | Bug because the current situation leads to (silent) failures on quite elementary file operations (like ftruncate(), chown(), chgrp()) on stream wrappers. |
|---|---|
| Issue priority | Major because we are talking about elementary file operations that fail and often fail silently. |
| Prioritized changes | This is a prioritized change because:
|
| Disruption | Not disruptive for core, contributed or custom modules: see comment #34. |
| Comment | File | Size | Author |
|---|---|---|---|
| #41 | interdiff.txt | 442 bytes | fietserwin |
| #41 | fully_conform_to_php5_4-2213241-41.patch | 13.62 KB | fietserwin |
Comments
Comment #1
fietserwinSome code for tests:
Results in:
And these are not warnings that can be ignored: the chmod is not performed and a curl_exec() will really fail later on.
Comment #2
fietserwinComment #3
fietserwinAnother issue to decide on: should stream wrappers clear the statcache or is that the responsibility of the using code. See comments #29 to #34 of parent issue #2107287: PHP 5.4 calls a new stream_metadata() method on stream wrappers not implemented by Drupal.
Comment #4
sunI'm switching the parent/child relationship around, since this issue pretty much is a nice meta to spawn child issues from.
Comment #5
sunHere's what I did — essentially to replace the issue summary with hard facts:
The attached patch-serial contains exactly that. The "interdiff" contains only 2).
For 2), I manually copied the streamWrapper class prototype into a file, removed illegal type-hints,
__construct(), and__destruct(), and added "function" to the declared methods.Aside from some argument variable name mismatches, the differences are:
Comment #7
sunAttached patch performs the necessary changes.
Including a full patch serial once more for the sake of reviewing the individual steps.
Comment #8
fietserwinThis is a review of the changes to interface PhpStreamWrapperInterface only. I will try to review the rest later.
The "see also" section is not copied on the other methods added (e.g. @see fseek on stream_seek()). Shall we leave that section out everywhere and only add the @see with a link to the copied doc on php.net?
Do we document this as such in Drupal?
Comment #9
sunfseek()).Comment #10
sunComment #11
sunCurious, were you able to review this further? :)
Comment #12
fietserwinI finally got some time to further review...
LocalStream:
If LocalStream is meant as the base class for all "Drupal special locations" wrappers (public, private, temp, and, yet in contrib, module, theme, library) then it should fully implement the PhpStreamWrapperInterface with real implementations, no "silent failures" by just returning false. IMO that is what this issue is all about.
Thus the patch should contain "real implementations" for:
- stream_cast: return $this->handle (see e.g. comment #1).
- stream_set_option: I think it should call one of: stream_set_blocking(), stream_set_timeout() or stream_set_write_buffer().
- stream_truncate: it should call ftruncate($this->handle).
- stream_metadata should also handle the options PHP_STREAM_META_OWNER_NAME, PHP_STREAM_META_OWNER, PHP_STREAM_META_GROUP_NAME, PHP_STREAM_META_GROUP.
Adding real functionality, also means writing new tests or adapting existing ones that just tested that false was returned (if these exist). We can do so method by method in separate child issues, but I don't think it will be to much for 1 issue.
ReadOnlyStream:
What is the use of this class? It is not subclassed in core, PHPStorm not even finds a dummy test class that derives from this! Could this be some kind of historical left over, made redundant by (the IMO better named) LocalReadOnlyStream?
I think that we can remove this class!
LocalReadOnlyStream:
This class should also override methods stream_cast and stream_set_option and trigger an error, like is done with stream_truncate. I guess there should be tests for these as well (including stream_truncate).
In general:
- I would like to see the same method order in LocalStream and LocalReadOnlyStream as in PhpStreamWrapperInterface, being the order as documented on php.net, but this is better done in a child issue.
Comment #13
sunHm. I don't think I'm going to be able to pull off the full list of desired changes in #12. That said, some replies:
stream_cast(): I checked various implementations on the net, and in fact, I wasn't able to find one that actually implements this method. I don't know whether returning$this->handleis correct?stream_set_option(): I wondered whether those stream_* functions wouldn't cause an infinite recursion?stream_truncate(): Makes sense.stream_metadata(): While chown()/chgrp() operations can be implemented, they cannot be supported for LocalStream, because they require the PHP process to be executed as a super-user (which you obviously don't want to do). They can also not be tested, since tests would fail for the same reason (unless tests are executed as super-user, which is definitely not recommended).ReadOnlyStream: Indeed looks unused, but that is not to be discussed here.LocalReadOnlyStream: Sorry for missing those.Lastly, adding new tests will require someone who is (much) more familiar with these (very) advanced filesystem/stream operations — in most cases, I wouldn't even be able to figure out what the actual expectation to assert is ;-)
Comment #14
fietserwinIf I find some time, I will give it a try but will first assign the issue to me then. So, in the mean time, everyone, feel free to assign it to you and give it a try.
- For stream_cast I have a "test"/expected behavior in #1
- stream_set_option: let's find out.
- stream_metadata: if options cannot or should not be supported we need to trigger an error (I thought that chown/chgrp can be done by the owner of a file without further rights, but I am no *nix expert).
- ReadOnlyStream: created #2236163: Remove class ReadOnlyStream (or turn into a trait?).
- LocalReadOnlyStream: should be done in the next patch.
Comment #15
sunComment #16
fietserwinIn addition to patch of #15:
- Some tests.
- Solved an error: stream_set_option() on read-only stream should only fail for option STREAM_OPTION_WRITE_BUFFER.
I am still not sure what to do with ReadOnlyStream. Implementing stream_cast() should be easy, but implementing stream_set_option() is difficult without knowing that there is a parent class that implements it full blown, even if we would do that in a trait that would be a problem.
Comment #18
fietserwinOK, I can't get any of the options in stream_set_option to work, at least not on Windows, even not on standard local files using the mentioned functions stream_set_timeout(), etc. If you search on the internet, there is some mentioning of this, but it is nowhere explicitly stated that it indeed does not work with local files on Windows.
As these are advanced use cases anyway, I suggest to not implement it on our local stream wrappers and fail with a clear error message saying so.
Comment #20
fietserwin- the PHP core function stream_set_write_buffer() returns -1, not false.
- improved some comments.
Comment #21
fietserwinI reviewed the parts that sun provided (in #15) and can give an RTBC on that.
If someone else can review my changes made on top of #15, we can progress. To that end, I tried to create some kind of interdiff between the patches in #15 and #20. In short, the differences between #15 and #20:
- Added some tests for those methods that got a real implementation.
- Undid the implementation of stream_set_option() including the override in the local read-only stream, as it could not be made to work (on Windows). Note that this implementation was added by sun on my request, but now - based on testing - I propose to not implement it any more.
Another note: ReadOnlyStream is not kept up to date with the changes, as we might remove that class (or update it in accordance with this issue) in #2236163: Remove class ReadOnlyStream (or turn into a trait?).
Comment #24
yched commentedLooks like this was really close to RTBC - anyone feels like picking that up ?
An update of the issue summary to clarify the actual API break impact (if any) would help evaluate if it's still time to get this in D8.
Comment #25
fietserwinRerolled, let's see if this patch does apply. also updated the issue summary.
Comment #27
fietserwinIssue #1676910: Rename randomName() to randomMachineName() renamed method TestBase::randomName to randomMachineName...
Comment #28
fietserwinComment #29
fietserwinComment #30
fietserwinLet's try to activate the testbot.
Comment #31
yched commented@fietserwin: Thanks !
Before putting too much work in there, I think the first thing that is needed now is a clear list of the exact API changes that would occur here, so that it is possible to evaluate the exact impact in terms of BC break (what kind of existing code would break, whether such code is likely to be widespread already or only limited to a couple very rare/specific use cases, whether there are ways to limit the amount of those breaks by adding BC layers...).
That would help core maintainers decide whether it's still time for this to get in D8 at this point.
Comment #32
fietserwinUpdated the issue summary as per #31. Most noteworthy change:
The actual API changes are minimal and exists of additions only: addition of some methods and addition of some optional parameters.
However, stream wrapper methods are normally not called directly, but via the standard available PHP file functions (fread, ftruncate, chmod, etc.). So the importance of this issue is not in the API changes but in correcting a broken file handling system.
Comment #33
yched commented@fietserwin: ok - "new methods added to PhpStreamWrapperInterface" means all exising contrib/custom stream wrappers would break, since they wouldn't implement the new new methods.
AFAICT, seems targetted enough to still be an acceptable break at this point - not my decision though :-)
Comment #34
fietserwinRE #33:
In that sense you are right, however:
Known contrib modules:
So, I don't see foresee any problems.
I hope to have clarified the reasons behind and the importance of this issue, so please review, so that a core maintainer can have its say on it.
Comment #35
yched commentedNot that I'm very familiar with this area, but reviewed #27 and didn't find much to complain about :-)
"verify that not too many restrictions are triggered" is a bit vague :-)
The phpdoc should be just @inheritdoc, and the reasoning about why we don't support it could be in a comment inside the method body ?
Also, we (:-p) generally try to avoid "we" / "you" in code comments.
More generally, since this expands the methods phpdocs in PhpStreamWrapperInterface, I guess there is still work to:
- expand those that are still just "@return int"
- unify whether or not we want to have links like "@see http://php.net/manual/streamwrapper.stream-seek.php" on each individual method
- remove existing docblocks and unify on {@inheritdoc} in subclasses
I guess this doc cleanup could happen in a followup, since the current PhpStreamWrapperInterface is pretty lame in terms of phpdocs (to say the least), and fixing that is not strictly speaking in the scope of the issue here.
Comment #36
fietserwinThanks for reviewing.
re #35.1: done, I just renamed the test method to one that better describes what is tested.
re #35.2: I think that users/overriders should be able to find out what a method does by only looking at the docs, thus without a need to read the code itself, even when that is available. So I left it as is, but did made a few changes to get a 1 line short description and removing the "we/you".
re #35.we/expand/unify/remove:
There already is a follow-up specific for the documentation: #2228087: PhpStreamWrapperInterface lacks docblocks. So
- I left the other usages of we, as they were already there.
- I left the "@return int" doc blocks, see comment #9.2.
- I left the @see tags, as they come from the PHP manual.
- I left any duplication, as there is a follow-up.
Though not completely following your suggestions, I hope that with my reasoning behind that, you can accept this patch as is.
Comment #37
yched commentedOk then - last nitpick :
I think our practice in those cases is something like :
Also, the language can be refined a bit IMO. Suggestion :
"The method is not supported on local files, since Windows systems do not allow it, and it is not needed for most usecases anyway. If needed, custom subclasses can provide OS-specific implementations for advanced use cases."
Other than that, looks RTBC to me ?
Comment #38
fietserwinAccording to the Drupal documentation guidelines an {@inheritdoc} must be the only line in a docblock (https://www.drupal.org/node/1354#inheritdoc), even when the phpdoc site does describe the construct you mention (but does allow so only for class docblocks...). As I also prefer to augment rather then repeat and extent, I go with your suggestion as well as the rephrasing (but augmented that one with specifically mentioning the return value false and the triggered error).
Comment #39
yched commentedThanks ! This looks good to me.
Comment #40
alexpottWhat's this?
For example...
Comment #41
fietserwin@alexpott: thanks for looking at it.
re #40.1: I updated the issue summary. I hope I understood the template correctly and this is what you were looking for.
re #40.2: This is a property of the proposed PHP stream wrapper class and thus what we should implement. However, as properties do not belong to interfaces, I removed it from the patch. Whether the local stream should implement it is unclear to me. Looking at where contexts are used, I only see http, ftp, curl and such usages, no local stream wrappers. Moreover, no errors or warnings are filed in the issue queue about PHP complaining about this.
re #40.3: I can do it here, but would like to postpone it to the child issue #2228087: PhpStreamWrapperInterface lacks docblocks (also see comment #36).
Comment #42
almaudoh commentedCore committer feedback has been addressed in #41. Back to RTBC per #39.
Comment #43
alexpottWhilst reviewing this I've notice that
Drupal\Core\StreamWrapper\ReadOnlyStreamhas a call to$this->getLocalPath();inReadOnlyStream::stream_open(). At the very least there should be an abstract implementation so that anything that extendsDrupal\Core\StreamWrapper\ReadOnlyStreamknows they have to implement this. All of this is another issue of course. Just documenting this here so someone might follow this up.The change from $uri to $path has made the existing documentation incorrect. I've fixed this on commit.
Committed 57291a5 and pushed to 8.0.x. Thanks!
Thanks for adding the beta evaluation for to the issue summary.
Comment #45
fietserwin@alexpott: thanks for committing. I hope to find some time to update the documentation as per #2228087: PhpStreamWrapperInterface lacks docblocks.
Regarding the ReadOnlyStream: IMO that's a class that should not be there, therefore I started: #2236163: Remove class ReadOnlyStream (or turn into a trait?).