File target fails when the URL (e.g. one provided by an API) redirects to a file, rather than being the direct URL of a file resource [#3126031]

Some APIs provide resource URLs that do not point directly to a file, but instead redirect the user to the file. For example, https://www.instagram.com/p/B9vy5EdgZX1/media/?size=l is proviuded by the Instagram API, but this actually redirects to https://scontent-lht6-1.cdninstagram.com/v/t51.2885-15/fr/e15/p1080x1080/89738651_155780038891572_5224457300248488192_n.jpg?_nc_ht=scontent-lht6-1.cdninstagram.com&_nc_cat=100&_nc_ohc=rGVubVRCFoAAX8YKjNO&oh=76c6ac182e0432bfffce7539e372631d&oe=5EB77DAB
This redirect causes issues for \Drupal\feeds\Feeds\Target\File::getFileName which expects to get a URI that contains a file extension that it can pass to \Drupal\Core\File\FileSystem::basename
One fix would be to make a request to $url as part of \Drupal\feeds\Feeds\Target\File::getFileName and update the value of $url whenever a redirect takes place.

Patch incoming :)

Comment	File	Size	Author
#12	file-target-fails-when-the-url-redirects-to-a-file-3126031-12.patch	2.68 KB	lesleyfernandes
#12	8.x-3.x: PHP 7.1 & MySQL 5.5, D8.9.20 512 pass
#11	file-target-fails-when-the-url-redirects-to-a-file-3126031-11.patch	1.65 KB	damondt
#11	8.x-3.x: PHP 7.1 & MySQL 5.5, D8.9 429 pass
#10	file-target-fails-when-the-url-redirects-to-a-file-3126031-10.patch	3.24 KB	damondt
#10	8.x-3.x: PHP 7.1 & MySQL 5.5, D8.9 417 pass, 2 fail
#5	file-target-fails-when-the-url-redirects-to-a-file-3126031-3.patch	1.23 KB	Phil Wolstenholme
#5	8.x-3.x: PHP 5.5 & MySQL 5.5, D8.7 374 pass, 5 fail
#3	file-target-fails-when-the-url-redirects-to-a-file-3126031-2.patch	966 bytes	Phil Wolstenholme
#3	8.x-3.x: PHP 5.5 & MySQL 5.5, D8.7 374 pass, 5 fail

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

8 April 2020 at 13:45

Phil Wolstenholme created an issue. See original summary.

Comment #2

Phil Wolstenholme CreditAttribution: Phil Wolstenholme as a volunteer commented 8 April 2020 at 13:45

Title:

File target fails when the URL contains a redirect to a file rather than being a direct URL of a file resource

» File target fails when the URL (e.g. one provided by an API) redirects to a file, rather than being the direct URL of a file resource

Comment #3

Phil Wolstenholme CreditAttribution: Phil Wolstenholme as a volunteer commented 8 April 2020 at 13:54

File	Size
file-target-fails-when-the-url-redirects-to-a-file-3126031-2.patch	966 bytes
8.x-3.x: PHP 5.5 & MySQL 5.5, D8.7 374 pass, 5 fail

Here's a rough patch that uses Guzzle to work out the 'final' URL of a resource after any redirects.

It also includes the changes proposed in https://www.drupal.org/project/feeds/issues/2977565#comment-13262487 to prevent the failures caused by query strings in a URL (another common thing with image resources returned by APIs).

Comment #4

Phil Wolstenholme CreditAttribution: Phil Wolstenholme as a volunteer commented 8 April 2020 at 13:54

Status:

Active

» Needs review

Comment #5

Phil Wolstenholme CreditAttribution: Phil Wolstenholme as a volunteer commented 8 April 2020 at 13:59

File	Size
file-target-fails-when-the-url-redirects-to-a-file-3126031-3.patch	1.23 KB
8.x-3.x: PHP 5.5 & MySQL 5.5, D8.7 374 pass, 5 fail

1 file was hidden/shown/deleted

File	Size
file-target-fails-when-the-url-redirects-to-a-file-3126031-2.patch	966 bytes
8.x-3.x: PHP 5.5 & MySQL 5.5, D8.7 374 pass, 5 fail

Here's an updated patch that adds the missing use statement for use GuzzleHttp\TransferStats;

Comment #6

8 April 2020 at 14:00

The last submitted patch, 3: file-target-fails-when-the-url-redirects-to-a-file-3126031-2.patch, failed testing. View results

Comment #7

8 April 2020 at 14:07

Status:

Needs review

» Needs work

The last submitted patch, 5: file-target-fails-when-the-url-redirects-to-a-file-3126031-3.patch, failed testing. View results

File	Size
file-target-fails-when-the-url-redirects-to-a-file-3126031-10.patch	3.24 KB
8.x-3.x: PHP 7.1 & MySQL 5.5, D8.9 417 pass, 2 fail

I took a slightly different approach putting the redirect calculation in getFile(). I also made it a selectable option in the target config form because of the performance impact it has.