Cache result of HTTP source on file system

This patch implements the cache class FeedsHTTPCache and also include a test that purely tests this class.

The cache class is not integrated with the rest of the Feeds code yet. I just first wanted to focus on the class itself.

Setting to "Needs review" for the testbot, but the work here is not done yet.

Try again.

This implements the case of expired cache entries.

Great, the cache class seems to do everything now that it is supposed to do.

Now adding all the stuff from #2032333-20: [Meta] Store remote files in the file system instead of in the database back, though without the changes to the fetchers that focussed on solving #2829097: Don't store raw source in feeds_source table. Should fail, as I haven't updated the tests in FeedsFileHTTPTestCase yet (that still assume that the complete cache entry is cached on the file system). I wonder though if changes are needed in the fetcher classes and/or http_request_get() to fix this issue. Lets find out.

Hm. Response code is expected to be cached as well. Made changes for this in FeedsHTTPCacheItem class.
Also updated the tests, though not checked yet if I did that properly.

I remember the reason why I decided to convert the HTTP response to a classed object when caching it: so that whenever $result->data was requested, the class could return the data from the cached file (by implementing a magic getter).
Now we are going to see if that works.

Almost no test failures!

I see that in #8 I had removed one line too much. That may be just everything?

+++ b/tests/feeds_fetcher_http.test
@@ -167,15 +159,11 @@ class FeedsFileHTTPTestCase extends FeedsWebTestCase {
-    $csv = implode("\n", $lines);

I have been testing the patch myself and I found the following issues:

1. When deleting a cached file manually, the content didn't get refetched when the HTTP source returned a 304. Even worse, files mapped to a file mapper lead to a broken link in this case.
2. I realized that when a different cache class is used, Memcache or Redis for example, the raw data would still be cached, as now purely the cache class 'FeedsHTTPCache' takes care that the raw data isn't saved to cache. But when a different cache system is used, FeedsHTTPCache is bypassed. So saving the raw data to a file should happen one step earlier.
3. When cache entries are deleted manually from the database, the files will remain and wouldn't get cleaned up. This would also be an issue when a different cache class is used (and the previous noted issue is fixed).

Attached a patch that add tests for the first two issues found. I'm already working on the fixes for these two, but I want to make sure that these tests are failing now. For the third issue I still need to think about how that fix and test that.

Oops. There was still debug code in the test.

Hm. The test FeedsFileHTTPTestCase::testHTTPCacheOverride() is still passing, while it was meant to fail.
New try.

Great, now FeedsFileHTTPTestCase::testHTTPCacheOverride() does fail. The attached patch should fix the following cases:

1. A manually removed cached file will not lead to errors: the source will be refetched.
2. When using an other cache class, the raw data fetched via HTTP will still be cached on the file system.

I changed the FeedsHTTPCacheItem class quite a bit. Now it knows the cache ID it will be saved under and it talks with the FeedsHTTPCache class about where to store the cached file (even when a different cache class is used).

The following problem still needs to be fixed: when cache entries are deleted manually from the database or when a different cache class is used, the files will remain and wouldn't get cleaned up.

Oops. The method FeedsHTTPCacheItem::saveResponseData() should call FeedsHTTPCacheItem::getCacheObject() to receive the cache object.

Here is a test that checks if orphaned cached files are cleaned up. Cache files can get orphaned when cache entries are deleted manually from the database or when a different cache class is used for the cache bin 'cache_feeds_http'.

Test should fail.

And here is a possible fix for the test failure. Adds quite a lot of code though.

In short:

• A queue called 'feeds_sync_cache_feeds_http' is added.
• Every 6 hours a job is started to scan the cache file directory for orphaned files. The job is not started if there are still items in the 'feeds_sync_cache_feeds_http' queue. The interval is configurable via a variable called 'feeds_sync_cache_feeds_http_interval' (documented in the README-file).
• For at max 50 files per queue item it is checked if there are still cache entries for these files. Files for which no cache entry exist are removed.

Let's see if this passes tests.

I think this is ready.

I found a small bug when manually testing cleaning up orphaned cache files. If no feeds needed be rescheduled, the cleanup task was not started.

Also added a test for this cleanup task.

I think the test FeedsFileHTTPTestCase::testCachedFilesCleanupOnHTTPCacheOverride() failed because cron already ran earlier, causing the variable 'feeds_sync_cache_feeds_http_last_check' to be set and not scheduling the cleanup task on the next cron run (as the cleanup task is only scheduled every six hours). The fix for this: empty the 'feeds_sync_cache_feeds_http_last_check' before running cron.

This patch adds a check for if the Feeds cache directory is writable. Also included tests that test the case of a non-writable cache directory. I have set the non-writable dir to 'file://non-writeable-dir/feeds' as that resulted into something non-writable locally. Not sure if that counts too for the testbot.

If the passes tests, I commit the patch.

Committed #29!