I've got some uploaded files with 'weird' characters in the filenames, for example: filename(version 2).jpg.

I can upload this file and view it via Drupal, use image styles on it etc, however, it's not served via the farfuture cdn feature.

This is because the farfuture code checks to see if the file exists, but does so based on the URL encoded version of the file path, however, it's not URL encoded on my filesystem.

FileUrlGenerator::generate is where the file_exists is called, but this might want to be fixed in ::getRelativeUrl because that seems to be where the code knows it's coming from a stream wrapper (and generates the local path).

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Steven Jones created an issue. See original summary.

Wim Leers’s picture

Assigned: Unassigned » Wim Leers

Thanks for the crystal-clear bug report!

\Drupal\Tests\cdn\Unit\File\FileUrlGeneratorTest::testGenerate() is testing "special" file names already. But \Drupal\Tests\cdn\Functional\CdnIntegrationTest::testFarfuture() isn't yet. So we should be able to expand that test to trigger a failure.

Wim Leers’s picture

Issue tags: +Needs tests
Wim Leers’s picture

I was wrong in #1, \Drupal\Tests\cdn\Unit\File\FileUrlGeneratorTest::testGenerate() only tests non-far-future cases. It's \Drupal\Tests\cdn\Unit\File\FileUrlGeneratorTest::testGenerateFarfuture() that we should expand to test a "special" file name, because that contains the bug described in the last paragraph of the IS.

But on top of that, we also should expand \Drupal\Tests\cdn\Functional\CdnIntegrationTest::testFarfuture(), but we should convert it to a proper integration test.

Wim Leers’s picture

Status: Active » Needs review
FileSize
1.28 KB

First, let's fix the imperfect simulation of how PublicStream behaves. We have to mock that, because PublicStream can only ever be actually available in a kernel or functional test, whereas this is a unit test (which we want, because it's much faster).

Status: Needs review » Needs work

The last submitted patch, 5: 2925507-5.patch, failed testing. View results

Wim Leers’s picture

FileSize
1.32 KB
2.9 KB

Voila, this shows that our unit tests weren't reflecting reality. That's of course the danger of unit tests. But in this case, they're worth the trouble. This is (or should be) the only flaw in them.

Wim Leers’s picture

Status: Needs work » Needs review
Wim Leers’s picture

FileSize
2.81 KB
5.38 KB

Now that our unit tests do reflect reality, we can expand it with the missing unit test coverage.

This should fail.

Status: Needs review » Needs work

The last submitted patch, 9: 2925507-9.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Wim Leers’s picture

This is the failure:

--- Expected
+++ Actual
@@ @@
-//cdn.example.com/cdn/farfuture/mklcqDAZef74DPzCW53N_mqaPMip9uNAKsluggC1sJk/1511362523/sites/default/files/llama%20%28aVD8XdYb%29.jpg
+//cdn.example.com/sites/default/files/llama%20%28aVD8XdYb%29.jpg

We expected the file to be served using a farfuture URL, but that was not the case. This is exactly the bug that was reported. So now we have a regression test 👍

Wim Leers’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
FileSize
4.36 KB
5.38 KB

And this then finally updates the integration test coverage. Instead of testing with public://druplicon.png and core/misc/drupal.js, this changes that all to public://druplicon ❤️.png.

This should cause another failure. And this means the regression test coverage is now complete.

Status: Needs review » Needs work

The last submitted patch, 12: 2925507-9.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
9.69 KB

Oops, I re-uploaded the #9 patch in #12. The interdiff was right though.

Status: Needs review » Needs work

The last submitted patch, 14: 2925507-12.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Wim Leers’s picture

Status: Needs work » Needs review
Related issues: +#974350: Far Future setting for Origin Pull mode
FileSize
2.94 KB
12.59 KB

Great, those failures were expected! Now let's fix it.

The root cause lies in using relative URLs as paths, but relative URLs may look like paths, but they are of course URL-encoded. And that's what I overlooked here. I got this right when I ported farfuture functionality from D6 to D7 in #974350: Far Future setting for Origin Pull mode, but I got it wrong in the D8 port. My bad.

Wim Leers’s picture

Assigned: Wim Leers » Unassigned
Priority: Normal » Major
FileSize
5.16 KB
14.13 KB

Green, yay!

And now we can actually clean up the logic a little bit (because with the few added lines, it looks weird). Slightly out of scope, but doing this in a separate issue also seems silly. Also fixing the 2 coding standards violations that I accidentally introduced earlier.

Also, this definitely is a major bug. Nothing actually breaks, which is why it's not critical, and which is why it went unnoticed until now!

Wim Leers’s picture

Can you please apply the patch, and confirm that it solves the problem? Thank you!

Wim Leers’s picture

Title: URL encoding not correctly handled by farfuture controller » URL encoding not correctly handled by farfuture functionality
Wim Leers’s picture

FileSize
1.76 KB
14.16 KB

Self-review.

  1. +++ b/tests/src/Unit/File/FileUrlGeneratorTest.php
    @@ -138,19 +139,29 @@ class FileUrlGeneratorTest extends UnitTestCase {
    +    // Generate file for testing non-shipped file.
    

    s/non-shipped/managed/

  2. +++ b/tests/src/Unit/File/FileUrlGeneratorTest.php
    @@ -138,19 +139,29 @@ class FileUrlGeneratorTest extends UnitTestCase {
    +    file_put_contents($llama_jpg_filepath, $this->randomMachineName());
    

    We should delete this file.

  3. +++ b/tests/src/Unit/File/FileUrlGeneratorTest.php
    @@ -138,19 +139,29 @@ class FileUrlGeneratorTest extends UnitTestCase {
         $drupal_js_mtime = filemtime($this->root . '/core/misc/drupal.js');
         $drupal_js_security_token = Crypt::hmacBase64($drupal_js_mtime . '/core/misc/drupal.js', static::$privateKey . Settings::getHashSalt());
    

    This code is unnecessary, we should delete it.

Wim Leers’s picture

Credited Steven Jones.

Steven Jones’s picture

Status: Needs review » Reviewed & tested by the community

Seems to work nicely!

  • Wim Leers committed ad8b62b on 8.x-3.x
    Issue #2925507 by Wim Leers, Steven Jones: URL encoding not correctly...
Wim Leers’s picture

Status: Reviewed & tested by the community » Fixed

Great, thank you for testing!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Wim Leers’s picture