I have a target alter callback that fetches file contents from a URL using FeedsEnclosure::getContent() and another that uses FeedsEnclosure::getFile() which in turn calls FeedsEnclosure::getContent(). When using drush queue-run feeds_source_import these both result in timeouts for files larger than around 2MB.

FeedsEnclosure::getContent() calls http_request_get which uses CURL. CURL timeout settings in http_request_get default to the variable http_request_timeout or 30 seconds introduced in #1480902: Are there timeouts on Feed importers?, which also introduced the ability to set a custom timeout value for individual importers. FeedsEnclosure::getContent() does not currently pass a timeout value to http_request_timeout, therefore it always uses the http_request_timeout variable or 30 seconds.

Proposed resolution

Note: quickest solution would be to increase http_request_timeout variable. However, this issue is intended to highlight that individual importers have their own timeout settings and perhaps FeedsEnclosure could make use of those settings when calling http_request_get.

If I have understood the problem correctly then in my case I think the ideal solution would be for FeedsEnclosure::getContent() to make use of the custom timeout setting of the current importer fetcher. An alternative solution might be to allow modules to set a timeout value, either by passing it directly to FeedsEnclosure::getContent() or FeedsEnclosure::getFile() or by adding a timeout property to FeedsEnclosure that can be set when instantiating FeedsEnclosure and then used by FeedsEnclosure::getContent().

#1480902: Are there timeouts on Feed importers?
#1048810: http fetch request... timing out.


lwalley’s picture

Attached is a patch that adds a timeout property to FeedsEnclosure that can then be used when instantiating FeedsEnclosure in a target alter e.g. new FeedsEnclosure($url, file_get_mimetype($url), $request_timeout);

In my case I wanted to make use of the importer's custom timeout setting so in my target alter I used $request_timeout = $source->importer()->fetcher->config['request_timeout'];

There is probably a better way to do this, but this seems to work for my immediate needs which is to increase the timeout when using FeedsEnclosure for particular importers.

twistor’s picture

Status:Active» Needs work

That looks like a good start.

Could we also add a setter for the timeout, as well as, actually use it in the mappers?

lwalley’s picture

Thanks for the quick response! I think what you're suggesting is remove $timeout from the constructor and instead add FeedsEnclosure::setTimeout public method so that in my target alter I would then revert to using $var = new FeedsEnclosure($url, file_get_mimetype($url)) and add $var->setTimeout($request_timeout). If I interpreted that right then yeah that would definitely work, and attached is an updated patch.

lwalley’s picture

Issue summary:View changes

Added a note that users can simply increase http_request_timeout variable to get around this issue, but that the purpose of this ticket is to explore passing timeout values from FeedsEnclosure::getContent() to http_request_get.