Problem/Motivation
Using this module, the feed import processes and completes, but does not import the entire feed. The problem only occurs on large feeds that take multiple cron jobs. Testing reproduced issue on load-balanced environments (Pantheon). The following error message is produced:
The feed "XX" encountered an error when processing: Drupal\feeds\Exception\FileNotFoundException: File <em class="placeholder">/tmp/feeds_http_fetcherXxx</em> does not exist. in Drupal\feeds\Result\FetcherResult->checkFile() (line 58 of /code/web/modules/contrib/feeds/src/Result/FetcherResult.php).
Downloading the temp file from the Feed log (or sftp) shows the temp file contains the entire feed.
The issue has been narrowed down to this module due to it setting the temporary file location.
Steps to reproduce
- Use a Drupal site on a load-balanced environment.
- Start import (in background) of a feed that uses feeds_fetcher_post with over 1000 entries.
- Wait for cron to run a few times to complete import.
- Check feed log at /feed/XX/log to see how many entries were imported.
- Feed log shows less than 1000 entries imported, usually a number such as 300, 700 or 900.
- DBLog at /admin/reports/dblog?type%5B%5D=feeds shows error as above.
- Feed temp file can be downloaded and is viewable through sftp from the private://feeds/log/XXXXX/source/ location created by feeds module, but not from the /tmp/ location created by the feeds_fetcher_post module.
Proposed resolution
If feeds_fetcher_post would not override the temp file location of HttpFetcher in feeds module, then this issue can be fixed.
Remaining tasks
User interface changes
API changes
Data model changes
Issue fork feeds_fetcher_post-3481810
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
megachriz@maintainer
It would be best if HttpFetcherPost no longer needs to override the
fetch()method. But perhaps that requires code fromHttpFetcher::fetch()to be split up further. Let me know if you need any changes to that class to make the implementation for this module easier.Comment #3
megachrizSee also #3481787-3: Large feeds do not import entirely
Comment #5
sleitner commented@megachriz
HttpFetcher::fetch()does not pass the feed configuration to theHttpFetcher::get()function.Add the feed_configuration parameter:
protected function get($url, $sink, $cache_key = FALSE, array $options = [], array $feed_configuration = [])function call:
$response = $this->get($feed->getSource(), $sink, $cache_key, [], $feed->getConfigurationFor($this));Comment #7
sleitner commented@andreayaya MR1 should resolve the problem, but the
fetch()method is still overritten, for now.Comment #9
sleitner commentedfixed in 2.0.0-alpha2