Some hostings(like Acquia) have reverse proxy in front of apache and their settings can't be overriden for some unknown reason.
When we have huge migrations - they are dependent from max_execution_time and batch chunk calculated accordinly.
But reverse proxy can't wait and brings GATEWAY_TIMEOUT to batch API.
Here is a patch, that extends default migrate settings and adds ability to override timeout to smaller amount of time for ability to work with batch API at all.

CommentFileSizeAuthor
#4 reverse_proxy_timeout-2508171-4.patch560 bytespodarok
188.diff576 bytespodarok

Comments

podarok’s picture

Issue summary: View changes

Status: Needs review » Needs work

The last submitted patch, 188.diff, failed testing.

podarok’s picture

Issue summary: View changes
podarok’s picture

Status: Needs work » Needs review
StatusFileSize
new560 bytes

update upstream

Status: Needs review » Needs work

The last submitted patch, 4: reverse_proxy_timeout-2508171-4.patch, failed testing.

podarok’s picture

Status: Needs work » Needs review
Issue tags: +Needs manual testing, +Needs tests

Looks like tests are broken

mikeryan’s picture

Status: Needs review » Needs work

First off, if you have huge migrations to run, I strongly recommend reading https://www.drupal.org/node/1806824. Drush ftw.

I'm a little reluctant to add something to address such a narrow use case (if I'm not mistaken, you're bumping up max_execution_time to a large value to minimize batch overhead?). That being said, if you want to pursue this, a couple of comments:

  1. The test failure is telling you something - my best guess is that ini_get('max_execution_time') in the testbot is returning NULL and you need to handle it.
  2. If you're introducing a migrate-specific variable, its name should reflect that (e.g., 'migrate_batch_max_execution_time').
justindodge’s picture

I came across this issue on a particular AWS setup.
@mikeryan - I think the actual issue is that max_execution_time needs to be reduced, otherwise the batch job runs too long and the gateway times out before it can finish, so you never get to the next one.
Drush could solve this of course, but you don't to have a huge migration to encounter the problem - any migration via UI that has more than one batch would fail.

In my case it was easiest just to reduce php's max execution time via php.ini, since I had access to the web server but not the reverse proxy.