Don't cache_set full feed result in http_request.inc (or make optional)

Yes! In my case cache table have 2GB size! There is no reason to store full files! Only hash or headers etc.!
Subscribing!

I don't know why, but!
I clean all my caches(including http_requests caches).
And reimport my db!
Now after import feeds recreate all caches but without full data, table size now very little (~660b/row, not 600kb/row).
So as i see this issue fixed.
Version = "7.x-2.0-alpha5"

Hm, no, I do some mistake when test this issue!
Actually feeds save full fetched data(in my case images files) in cache table!

Sry for wrong info in #4

Hi all, I have taken the courtesy to create a patch for making the feed cache optional.

Our use case was a bit similar. We are running a feeds import every minute in order to sync two different systems. The file feeds is downloading is about 20MB in size, which cause issues with our memcache setup as memcache only allows 1 megabyte objects in size by default. In this case it tries to split it up to 20 different chunks which causes additional delay. For us in the end, we don't need caching as the file it downloads is dynamically generated and thus will always be different anyway.

I have created a patch for this issue, which adds an extra option in the feed fetcher configuration: "Cache HTTP result of request" (see screenshot). This is option is enabled by default in order to maintain backwards compatibility.

With this patch you will be able to disable the cache on a per-feed basis on the fly. It maintains full backwards compatibility with existing feeds.

Attaching screenshot of what the option looks like in the backend, and updating status to needs review.

@leendertdb
This is cool! I make this a child issue of #2032333: [Meta] Store remote files in the file system instead of in the database so that it becomes a target for Feeds 7.x-2.0-beta4.

Note though that your patch doesn't fix the whole issue. Image files for example (when mapping to an image field), are also downloaded via http_request_get(), but not via the HTTP fetcher. So these would still end up in the cache. See FeedsEnclosure::getContent() in feeds/plugins/FeedsParser.inc.

It would be great if you want to give #2829096: Cache result of HTTP source on file system a review too. The goal of that issue is to store the cached source on the file system and the metadata of the HTTP response in the database.

As I don't like the way options are added to http_request_get(), I opened #2877125: Refactor http_request_get().

Here is a reroll and an automated test.

Since with recent changes the downloaded HTTP result is now cached on the file system by default, I changed the description of the "Cache HTTP result of request" option to this:

Disabling this cache means that the downloaded source will not be cached (for example: on the file system, in the database or on memcache), but will be redownloaded on every feeds import attempt. This can be helpful if the source to download is dynamically generated and will always be different, or if it is very large (50 MB+) which could cause PHP memory timeout issues.

If you are having issues with the feeds importer not processing changes to the source file, or are experiencing caching issues, you can disable the caching of this feeds content.

I've done an experiment to optimize downloading data via HTTP, see #2880129: Optimize downloading data via HTTP. With the help of the "Display memory usage" setting from the Devel module and by executing the cron manually using the link on the Drupal status report page (admin/reports/status) I came to the conclusion that the real memory hog is in feeds_http_request() (formerly http_request_get()) and the memory usage barely increases after that (even with the optimization). Therefore claiming disabling the cache could fix PHP memory timeout issues is wrong.

In the new patch, the description is now this:

Disabling this cache means that the downloaded source will not be cached (for example: on the file system or on memcache), but will be redownloaded on every feeds import attempt. This can be helpful if the source to download is dynamically generated and will always be different, or if it is very large (50 MB+) which would cost additional webspace.

If you are having issues with the feeds importer not processing changes to the source file, or are experiencing caching issues, you can disable the caching of this feeds content.

Committed #11 with small changes in the wordings. The last bit now reads:

If you're having issues with Feeds not processing changes from the source file, or if you are experiencing caching issues, you can disable the caching of this feeds content.

@MegaChriz,

Thanks for committing the patch. One note though about the change in the form description: I see that you changed my wordings of "1-4MB" to "50MB+". The reason why I added the "1-4 MB" part is because of a limitation in memcache.

By default memcache only supports cache items with a size of 1 MB maximum. When e.g. you try to cache a 10MB variable it will split it up into 10 different chunks of 1MB. This can cause a lot of overhead with memcache sets/gets and also (depending on whether your memcache instance is on the local machine or a different node in the network) a lot of network bandwith usage.

The same reasoning applies to the database, when the database instance is on another machine in the network the database cache_gets and cache_sets can add up really fast where a 1gbit/sec internal network connection can be a bottleneck really fast. At least that's what was happening at our end with a multi server setup on a high traffic drupal platform.

Just a FYI about the reasoning behind my originally proposed description.

@leendertdb
I changed that text because in the dev version I have introduced a cache class for the cache_feeds_http bin. The new default is that only headers are cached in the database and the raw file data is cached on the file system. I've made this so that even when you set a different cache class for cache_feeds_http, the file contents are still cached on the filesystem. You can only explicitly override this behavior if you use a cache class that extends FeedsHTTPCache. This way the file contents should not end up in memcache when using memcache for the cache_feeds_http bin. I've backed up this behavior with an automated test. See FeedsFileHTTPTestCase::testHTTPCacheOverride().

@MegaChriz,

That sounds like a good change, would have solved our issue at that time as well. Thanks for clarifying. +1 :-).