CommentFileSizeAuthor
#24 interdiff.txt998 bytesslashrsm
#24 2650976_24.patch26.25 KBslashrsm
#23 interdiff-2650976-21-23.txt1010 bytesphenaproxima
#23 2650976-23.patch26.25 KBphenaproxima
#21 interdiff-2650976-19-21.txt589 bytesphenaproxima
#21 2650976-21.patch26.36 KBphenaproxima
#19 interdiff-2650976-16-19.txt7.76 KBphenaproxima
#19 2650976-19.patch26.35 KBphenaproxima
#17 2650976-16.patch23.68 KBphenaproxima
#15 interdiff-2650976-14-15.txt3.73 KBphenaproxima
#15 2650976-15.patch24.76 KBphenaproxima
#14 interdiff-2650976-12-14.txt4.53 KBphenaproxima
#14 2650976-14.patch22.36 KBphenaproxima
#12 interdiff-2650976-11-12.txt8.29 KBphenaproxima
#12 2650976-12.patch17.73 KBphenaproxima
#11 interdiff-2650976-9-11.txt389 bytesphenaproxima
#11 2650976-11.patch11.08 KBphenaproxima
#9 interdiff-2650976-5-9.txt3.24 KBphenaproxima
#9 2650976-9.patch10.6 KBphenaproxima
#5 interdiff-2650976-4-5.txt2.52 KBphenaproxima
#5 2650976-5.patch8.94 KBphenaproxima
#4 2650976-4.patch8.06 KBphenaproxima
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

phenaproxima created an issue. See original summary.

slashrsm’s picture

Issue tags: +Media Initiative, +D8Media

This looks amazing indeed! Just two nitpiks.

slashrsm’s picture

Status: Active » Needs work
phenaproxima’s picture

Status: Needs work » Needs review
FileSize
8.06 KB

Okay, I have finally come back to this and rolled a patch :) It could probably use a test or two, but I'd like to have the implementation OKed first before I do that.

phenaproxima’s picture

I was having trouble getting user avatars to be embedded directly in the tweet thumbnails -- this is now fixed.

samuel.mortenson’s picture

Some review:

  1. +  if ($variables['avatar']) {
    +    $data = file_get_contents($variables['avatar']);
    

    Would this make an external request every time the preprocess function is called? I'm guessing that once the SVG is generated this will never be called again, so that might be OK. Just checking!

  2. +      $variables['avatar'] = 'data:image/' . strtolower($extension) . ';base64,' . base64_encode($data);
    

    Due to security paranoia, it might be good to hard-core the image extension if we know what to expect from Twitter. For example, if the extension of the avatar is SVG, that SVG could contact executable code.

I also mentioned at NYCCamp that there is a fair use concern with making copies of Tweets. You can read Twitter's article on this here: https://support.twitter.com/articles/20171959. The TL;DR (IANAL) of the liability is that if I make a Tweet and you make a copy of it on your site that violates Fair Use, I can file a complaint against Twitter and might get your API access revoked. This sounds scary, but if these thumbnails are only displayed in the backend it's unlikely that you are going to get complaints filed against you (unless your content editors created a Tweet you've embedded). Even if a complaint does get filed against your account, if you're only using this in an admin interface (i.e. the thumbnails are not publicly accessible), you are probably fine.

phenaproxima’s picture

Would this make an external request every time the preprocess function is called? I'm guessing that once the SVG is generated this will never be called again, so that might be OK. Just checking!

The request is only made when the SVG is generated. From then on, the avatar is wholly embedded in the generated SVG and never needs to be downloaded again.

Due to security paranoia, it might be good to hard-core the image extension if we know what to expect from Twitter. For example, if the extension of the avatar is SVG, that SVG could contact executable code.

We don't know what to expect from Twitter, so we cannot hard-code the extension. I'm pretty sure Twitter does not allow SVGs in avatars, but if we want to err on the side of paranoia, we could probably validate the extension before the SVG is generated and ensure that it's gif, jpe?g, png, or webp. And if it is none of those extensions, omit the avatar from the generated thumbnail. That seems like a fair compromise to me.

Fair use is a tricky one. Obviously we cannot guarantee that people won't trample all over Twitter's fair use policy, so I suggest that we add a checkbox to the Twitter media type plugin's configuration that specifically enables the auto-generation of thumbnails. The checkbox could have an explanation about the risks (and a link to Twitter's policies) in its description. So that way, at least people would be informed about the hornets' nest they are potentially kicking. Drink responsibly.

samuel.mortenson’s picture

... but if we want to err on the side of paranoia, we could probably validate the extension before the SVG is generated ...

Yep, that would work for me. The specific exploits I can think of are 1) Embedding SVGs with untrusted data 2) Malformed extensions that could break out of the tag. I guess that we should trust Twitter to validate extensions, but the more checks we can do, the better!

... I suggest that we add a checkbox to the Twitter media type plugin's configuration that specifically enables the auto-generation of thumbnails ...

That, or any explicit documentation should be fine. I'm not sure how to properly warn users when enabling a module, so a check box makes sense since they have to interact with it.

phenaproxima’s picture

Implemented both ideas.

Status: Needs review » Needs work

The last submitted patch, 9: 2650976-9.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
11.08 KB
389 bytes

Sigh. Forgot to update config schema.

phenaproxima’s picture

After a quick discussion in the #drupal-media IRC channel, it was decided that this is really hard to test as long as the ability to fetch tweets from Twitter's API is not wrapped by a service (this functionality, after all, does not work without the Twitter API). To that end, I've moved the tweet-fetching stuff into its own service, the TweetFetcher.

mtodor’s picture

A bit tricky to test. Here are few ideas that maybe can help.

  1. create few fixtures files with exported result that you get from fetchTweet() execution
  2. then you can mock service to return saved fixture (depending of tweet id, so that you can test with few different tweets)
  3. there is also option to create test module -> and in that module override your service to do mocking as explained above -> then you can go with functional tests, I guess, instead of unit
phenaproxima’s picture

Okay, I wrote a test. It's not the most extensive coverage in the world but it should at least verify that thumbnails are generated when they should be. Every new code path I added in this patch (except for the preprocess function) is tested here.

phenaproxima’s picture

Fixed an invalid URL in TweetFetcher::fetchTweet() -- this would never be catchable in the tests because it would only show up when using Twitter's real API -- and added the Twitter logo as a background to the generated thumbnails. Sorry about the extra cruft in the interdiff; that's the result of re-rolling this patch against HEAD following typo fixes.

Status: Needs review » Needs work

The last submitted patch, 15: 2650976-15.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
23.68 KB

Okay, here is the actual patch without any cruft from 8.x-1.x in it. Sometimes I rock at git, and other times it eats me for lunch.

slashrsm’s picture

Status: Needs review » Needs work

Finally managed to look at this. Looks great. I have a few comments, but we should be almost there. Great work!

  1. +++ b/src/Plugin/MediaEntity/Type/Twitter.php
    @@ -138,21 +162,23 @@ class Twitter extends MediaTypeBase {
    +              $image_url = $this->getField($media, 'image');
    +              return file_unmanaged_save_data($image_url, $local_uri, FILE_EXISTS_REPLACE);
    

    $image_url is not the actual data... Yes, if this is broken then it is an existing bug, but would be nice to fix it anyway.

  2. +++ b/src/TweetFetcher.php
    @@ -0,0 +1,91 @@
    +
    +class TweetFetcher implements TweetFetcherInterface {
    

    Class docblock missing.

  3. +++ b/src/TweetFetcher.php
    @@ -0,0 +1,91 @@
    +    // Assume credentials have been set and query Twitter's API.
    +    $response = $this->twitter
    

    $this->twitter may not be initialized at this point.

  4. +++ b/src/TweetFetcher.php
    @@ -0,0 +1,91 @@
    +    // TODO: Check for errors in the response and throw errors if there are any.
    

    Any ideas for checks? If they are straightforward could we add then now?

  5. +++ b/src/TweetFetcher.php
    @@ -0,0 +1,91 @@
    +    // If we have a cache, store the response for future use.
    +    if ($this->cache) {
    +      $this->cache->set($id, $response);
    +    }
    

    Set a lifetime?

  6. +++ b/src/TweetFetcherInterface.php
    @@ -0,0 +1,47 @@
    +   * @throws \Exception
    

    Would be nice to have custom exception class.

  7. +++ b/tests/src/Kernel/ThumbnailTest.php
    @@ -0,0 +1,177 @@
    +/**
    + * @group media_entity_twitter
    + */
    +class ThumbnailTest extends KernelTestBase {
    

    Description missing.

  8. +++ b/tests/src/Kernel/ThumbnailTest.php
    @@ -0,0 +1,177 @@
    +    $uri = 'public://twitter-thumbnails/12345.ico';
    +    touch($uri);
    

    It seems that this assumes entity id. Is it safe to do that (usually not)?

  9. +++ b/src/Plugin/MediaEntity/Type/Twitter.php
    @@ -280,6 +322,40 @@ class Twitter extends MediaTypeBase {
    +    // Ensure the destination directory is writable. (TODO: Throw an exception
    +    // or otherwise take action if it isn't.)
    

    Let's do that :)

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
26.35 KB
7.76 KB

All fixed, except #8.

It seems that this assumes entity id. Is it safe to do that (usually not)?

That's a tweet ID, not an entity ID. I just didn't think I needed to bother with a 20-digit number in the test ;)

Status: Needs review » Needs work

The last submitted patch, 19: 2650976-19.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
26.36 KB
589 bytes

Man. Rename one little service, get smote by the testing gods. The life of a programmer is fraught with peril.

Status: Needs review » Needs work

The last submitted patch, 21: 2650976-21.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
26.25 KB
1010 bytes

Okay, testbot, you wanna play rough? Let's play.

slashrsm’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
26.25 KB
998 bytes

Fixed a nitpik. Should be ready to go.

  • slashrsm committed e5d7c69 on 8.x-1.x
    Issue #2650976 by phenaproxima, slashrsm: Use SVG to generate thumbnails...
slashrsm’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Needs work

The last submitted patch, 24: 2650976_24.patch, failed testing.

slashrsm’s picture

Status: Needs work » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.