Problem/Motivation

We are trying to endorse this module and use it in Varbase distro.
However, after assessing its functionality, and potential impact on performance, we noticed the following:

For each image generated, there will be two requests instead of one. One with 301 redirect, and the other is the image via the module's route.

  • Example:
    https://mysite.com/drimage/2500/1406/324/-/sites/default/files/2020-09/myimage.jpg which will do a 301 redirect to:
    https://mysite.com/drimage/2500/1406/324/- which will render the image with a Content-Type: image/jpeg header.

This by itself is problematic because it adds performance overhead (two requests instead of one), and also the image is not a "real" image with .jpg extension that can be served by the webserver directly. Instead, it's the Drupal route that requires Drupal to bootstrap. This means it will not make use of reverse-proxy cache rules for example.

Steps to reproduce

  1. Enable and configure the module
  2. Open Chrome Developer tools, Network tab, filter by images. And investigate the image requests

Proposed resolution

Looking at similar implementations, you'll find bbc.com for example doing a similar JS dynamic image generation. It's not a Drupal site, but the same concept applies

I don't have a specific way of how it should be done, but it is possible to apply this flow?

  1. Generate image on first request when image style is not generated
  2. Server the real generated image using it's real path (with .jpg or similar) without redirects, or bootstraping Drupal

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork drimage-3283460

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

  • 2.x Comparechanges, plain diff MR !13

Comments

Mohammed J. Razem created an issue. See original summary.

weseze’s picture

The reason is 2-fold:
1/ We wanted SEO friendly image URL's so we added the full filename. Drupal doesn't work well with this kind of URL so it is rewritten to a functional URL
2/ There is no way of generating the image without bootstrapping Drupal (if the image does not exist)

But, there is an optional system you can enable (only tested with apache webserver) that fixes this issue for every request but the first.

See the README section:

* (optional) If you are using an Apache webserver you can use custom htaccess
rewrite rules to lower the load on your server and have existing files be
loaded directly instead of bootstrapping Drupal.
Use this snippet in your composer file:
"extra": {
"drupal-scaffold": {
"file-mapping": {
"[web-root]/.htaccess": {
"prepend": "web/modules/contrib/drimage/htaccess.prepend.txt"
}
}
}
}

This should fix 99% of performance issues.

  • weseze committed a4376a0 on 2.x
    Issue #3283460 by weseze: 302 redirect instead of 301 to prevent 404 on...
  • weseze committed abbbe92 on 2.x
    Issue #3283460 by weseze: Fix documentation on htaccess.prepend.txt
    
rajab natshah’s picture

Thank you, Wesley for your time and for working on the nice and smart way of resizing images with drimage.

Noticed that most systems on the internet will use a physical image file link to get the image with a direct HTTP(S)/TCP protocol call on the asset-hosted server, with no further redirecting.

Some servers could manage redirecting in a quick way, but some http server like webpack and vite may still have issues while working or developing.

Hoping to change the image tag src attribute from the drimage url route to use a custom data-drimage-src-route attribute insted, with the drimage route url which fills in the right physical image url in the src attribute on observing resizeing or sizing of the page and containers on load or reload.

The Intersection Observer can manage that with all drimage media view modes, and on resize too
The Drupal native image styles system will manage to generate the image if it was not in the system with no further redirecting and recalling.

weseze’s picture

Status: Active » Needs work

Drupal can not create image styles dynamically based on the url. That's actually why we currently need the redirect. It ensures a valid image style is created.

An img-tag without a src attribute is invalid HTML (and very bad for SEO) so we shouldn't do that.

I do believe the current solution, although not perfect, is the best compromise to make this work in Drupal. (not considering the outdated JS ;))

If there are performance issues on webservers other then Apache, I'd be happy to accept further code-snippets/templates and documentation to get around them.
Like @Grayle did in #3248462: Allow apache to serve drimage styles from disk

rajab natshah’s picture

Thanks, Wesley
Having 2 weeks planned time on physical implementing and coding the use of drimage.

My points are:

  • No need to create a new "Image Style" config on every hit ( They can be a preset with Drimage settings with limitation options, and more settings are needed for sure)
  • The normal Drupal system will look at the image src and check if it was generated before or not ( if not it will generate it )

Changing the height in the drimage route for example to a height of 3000px
The system will do two things:
  • It keeps creating new image style configs for each change of height
  • It will generate a physical image file for that height too

For example:

  • A basic bash command with a for loop from 1 to 10000 on changing the height, which using wget or curl for the drimage route on a select one image will create 10000 new "Image Style" configs in the active config on the production server
  • A 10000 new physical images will be created on the server too

Proposed resolution

  • Have Minimum difference per image style on the height.
  • Have a set of configs for height in the module as it has for the width ( minim height, max height, step of change for height too )
  • Change the image tag src drimage url to use a custom data-drimage-src-route with the drimage route url which fills in the right physical image url on observing resize or sizing of the page and containers

Listed that in #3294297: Have Minimum difference per image style on the height like the width

weseze’s picture

No need to create a new "Image Style" config on every hit ( They can be a preset with Drimage settings with limitation options, and more settings are needed for sure)

Drimage does not do that, only the first time a request comes in for an image style that is not available yet will it do so. All subsequent requests will just use an existing image style (if one matches)
The "preset" thing you suggest by adding more settings is exactly what I tried to prevent with this module. If you know beforehand what image styles you are going to need to make your images responsive, then you are MUCH better off using normal responsive images from drupal core.

The intent of the module is that you simply do not need to think what sizes and image styles you might need in what circumstances to make your images responsive.
Again, if you are willing to do that (and you probably should ;)), use responsive images from drupal core. They will suit you much better.

The normal Drupal system will look at the image src and check if it was generated before or not ( if not it will generate it )

No argument here, drimage relies on this behaviour.

The whole extra drimage request and redirect is merely here to calculate and, if needed, create image styles on the fly. Part of this system has become obsolete because the JS also does the calculation... (it didn't to that in the early versions)
Perhaps we could do a rewrite where the JS does much more of the heavy lifting. But we will still need some sort of backend controller to generate image styles based on what the frontend wants.

Changing the height in the drimage route for example to a height of 3000px
The system will do two things:
It keeps creating new image style configs for each change of height
It will generate a physical image file for that height too

The option to prevent this is already implemented and is called "Maximum allowed ratio distortion".
Fixed pixel values would could massive differences in how distorted images can get, depending on their width. So I created a distortion calculation (in degrees/minutes) to allow for the same kind of distortion regardless of the width of an image.
A 200px height diff on a 320px wide image would be unacceptable, but on a 3200px wide image it would probably be ok.
The default value for this setting is set to 60 minutes (1 degree) and is intentionally very low to prevent distortion. You can probably get away with 10 degrees of distortion here. Please experiment with this value to reduce the amount of image styles being generated.
I'd be happy to change this default value based on your experience. (I personally don't use this option anymore)

Regardless we should provide a setting to completely disable this option (and do so on fresh installs) since it is obsolete now that we have image_widget_crop integration.

One final note: if you do change the drimage settings, you will have to delete all existing "drimage" image styles to get correct results. I don't think this has been clearly documented yet...

rajab natshah’s picture

Noted; Wesley
Thank you so much for having time to follow up on this issue.

The Drimage module idea is a genius idea which was implemented for Drupal developers by your team.

For sure you know more

Our teams are experimenting with options. Hoping to collaborate more to improve our products.

More ideas may come on:

  • The aspect ratios routing system, not a direct size routing system
  • The limitation system for max or min image styles
  • The HTML 5 intersection observer in the java script

I will leave them for now.
We may work on them later or have custom modules to test the change of logic and benchmark or performance

Managing with what the current module offer in round one.
Managing websites with Responsive Image Styles is complex and time consoming
Drimage is the right move from the old way in Drupal core. It is the easer option and most dynamic one

weseze’s picture

The aspect ratios routing system, not a direct size routing system

This can already be achieved by installing & configuring https://www.drupal.org/project/image_widget_crop
Once installed a new option will become available in the display settings for drimage.
Drimage integrates with the image widget cropping styles directly to provide aspect ratio scaling.

rajab natshah’s picture

Tested Drimage with Image Widget Crop options

Redirects are working with apache.


Crop of standard 16:9 was used in the media display
A route link of "/drimage/900/0/13/standard_16_9/sites/default/files/2021-08/startup-1.jpg"
will redirect with 302 to "/sites/default/files/styles/drimage_900_0_standard_16_9/public/2021-08/startup-1.jpg"
Crop of Square 1:1 was used in the media display:
A route link of "/drimage/500/0/3/square_1_1/sites/default/files/2021-08/coworking-5.jpg"
will redirect with 303 to "/sites/default/files/styles/drimage_500_0_square_1_1/public/2021-08/coworking-5.jpg"

Still, on changing "/drimage/900/0/13/standard_16_9" to "/drimage/900/100/13/standard_16_9", "/drimage/900/150/13/standard_16_9" or more options, the results from the new image style config and the image will follow with changes (I thought if an Image Widget Crop were used, the shape of the image will lock )

Making a benchmark round to study the lovely and smart "Maximum allowed ratio distortion" condition.


Another note in the documentation in code
https://git.drupalcode.org/project/drimage/-/commit/abbbe92#fda3484edf8d...
"prepend": "web/modules/contrib/webform/htaccess.prepend.txt"
should be changed to
"prepend": "web/modules/contrib/drimage/htaccess.prepend.txt"

  • weseze committed bf8a344 on 2.x
    Issue #3283460 by weseze, Rajab Natshah: Fix documentation on htaccess...
  • weseze committed fac407c on 2.x
    Issue #3283460 by weseze, Rajab Natshah: image_widget_crop styles should...
weseze’s picture

Fixed these issues.

rajab natshah’s picture

Thanks, Wesley

drimage 2.0.10 was released on 8 August 2022
https://www.drupal.org/project/drimage/releases/2.0.10

Having full testing round after the release

weseze’s picture

Status: Needs work » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

josebc made their first commit to this issue’s fork.

josebc’s picture

Version: 2.0.8 » 2.3.0
Assigned: Unassigned » josebc

Since this issue already exists I will add the changes here, initial PR for an alternative method of loading images similar to what core uses for image styles.
This solution changes the URLs to the physical path of the generated file and falls back to the router using a kernel event listener.

mohammed j. razem’s picture

Hoping for the maintainer(s) to reopen this issue.

We have created a fork of the module (https://www.drupal.org/project/drimage_improved) to demonstrate a fully functional module that uses an alternative image loading method similar to what core uses for image styles.

This forked module is largely similar to the current pull request (PR), with the main difference being the introduction of some install hooks to replace Drimage plugins as drop-in replacements.

Our intention is not to maintain or keep this fork if the PR is merged. We hope that this PR is merged with the improvement logic, which offers a significant performance boost compared to redirects and Drupal bootstrapped-image serving.

Once the PR is merged, we can shut down the fork and fully rely on the Drimage module.

weseze’s picture

Status: Closed (fixed) » Needs review

I'll reopen and try to make some time to review this. Sounds like a much better approach!

mohammed j. razem’s picture

Great! Happy to discuss if you have any questions.

weseze’s picture

Status: Needs review » Needs work

I did some small tests and have some feedback.
I will try and make some more time this week to further test and maybe try and get this fully working but feel free to help me out here ;)

1/
The first request causes a redirect. Is this something we can avoid?

2/
The image_widget_crop handling contains an error. See revised code below that does work correctly.

if ($this->moduleHandler->moduleExists('image_widget_crop') && isset($style_parts[3])) {
  $width = $style_parts[1];
  $height = $style_parts[2];
  // Need to implode all parts from index 3 and further to get the correct iwc_id.
  // The image widget crop id itself can contain underscores.
  $iwc_id = implode('_', array_slice($style_parts, 3));
}

3/
Do we need the same kind if checks for the focal_point integration?

4/
I think there is some code + documentation and example to rewrite url's via apache rewrite rules: this will need to change also?

5/
Is there any other code we can delete once this new delivery mechanism is in place?

rajab natshah’s picture

Fixed in the forked Drimage Improved project
#3452971: Redirect to 404 Page for Invalid Image Paths
Better to have a look at the latest changes at
https://git.drupalcode.org/project/drimage_improved

Thanks a lot for the idea of Drimage, BBC, CNN, and more are using this idea!

At this point we only can create MR from an issue fork, no direct project fork in Drupal.org

My only wish is that Drupal.org allow us to have a clean fork reference.
Like what we have in github.com or gitlab.com fork system.
to keep the forks related, as a child change of the original project, which could still upstream or merge pull requests, from any direction from forks or original.
weseze’s picture

Any possibility of updating the MR instead of forking?

I am willing to commit to get this working within drimage itself.
But comparing the forked module to the drimage, porting over changes and testing them, is not making this easy for me... :(

josebc’s picture

Hello @weseze
Regarding the redirect issue, I'm doing some work here https://git.drupalcode.org/issue/drimage_improved-3456741/-/compare/1.0.... but since there was a lot of functionality in the controller I ended up moving it to a new service.
Please have a look and let me know what you think if you can.

josebc’s picture

Assigned: josebc » Unassigned
Status: Needs work » Needs review
weseze’s picture

Thanks for the work! I'll make time at the end of the week and try and get it merged in and released.

trafo made their first commit to this issue’s fork.

trafo’s picture

Thanks for all the work.

I updated DrimageSubscriber to use rawurldecode() because image style url is generated via FileUrlGenerator that uses Drupal\Component\Utility\UrlHelper::encodePath() which uses rawurlencode() to encode path.

The difference is small, but if user decides to use + symbol in file name, it would be decoded as space instead of plus and this results in "Image not found".

ok4p1’s picture

What is left to get this merged?

weseze’s picture

I really need to review this, but it is low on my priority list... very sorry for that but it is the reality... :(

Could you provide feedback on the questions I asked in #22? That would at least help me determine if these changes would require a minor or major release, or if some more work is needed to work on backwards compatibility.

jarnetb made their first commit to this issue’s fork.