Provide better URL validation and error reporting for invalid feed URLs

I can't confirm this for http://developmentseed.org/rss.xml and beta8. Please reopen (and provide more detail) if problem persists on latest version.

I don't think this issue is related to feed xml formats. I've tried RSS0.93, RSS2.0, Atom 0.3 and some feedburned feeds, and I get the same error:

"Download of feed://rss.slashdot.org/Slashdot/slashdot failed with code -1003."

I think this is a related issue though: http://drupal.org/node/752862

I ran into the same error. I'd appreciate any advice.

here's the url for the feed I was having trouble with: feed://marketplace.publicradio.org/tools/rss/rundown_rss.php?show_id=14
I'm using Drupal 6.16 with feedburner 6.x-1.0-beta4

I am also getting the same error

An error has occurred.
Please continue to the error page
An error occurred. /batch?id=4&op=do
Fatal error: Call to undefined function curl_init() in /var/aegir/platforms/managingnews-1.0-beta9/profiles/managingnews/modules/contrib/extractor/libraries/placemaker.inc on line 28

I have tried to add this feed from drupal.org/planet:
http://drupal.org/planet/rss.xml

and this feed from google news
http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&q=drupal&cf=all&out...

Please provide your PHP version, platform, and confirm that curl is available.

#5: That error seems to indicate that you don't have cURL installed. Are you using Yahoo! Placemaker for term extraction? If so, please open a separate ticket, as that issue would likely be unrelated.

I'll update with version info in a little while. This problem went away when I found a simpler url for the same feed.

Hi Alex, I've been getting these errors, too. Here are a few feeds that don't seem to work:

feed://search.twitter.com/search.atom?q=microbicide
feed://www.avac.org/ht/a/GetFeedAction/content/Releases
http://www.eatg.org/rss/feed/news

What else would you like to know about them?
MN-1.0-beta9
Mac OSX 10.6.4
Safari 5.0.1

Thanks, Marilyn

Hi Marilyn,
Do any feeds work properly on your system?

The first two, which start with feed://, will not work because feed:// is an invalid protocol. It's something that Safari (which I see you are using) adds to the beginning of feeds for some reason. You must always change feed:// to http://.

I was able to import the third feed without errors. Please be sure you have curl installed for PHP.

It looks like this could be a case of poor URL validation and error reporting in Managing News.

In the original issue post at the top of this page, it looks like the protocol (http://) was omitted from the feed URL. In other situations, users are using feed:// as the protocol instead of http://.

Perhaps feeds should know to use http:// if the protocol is omitted or in the case that feed:// is used. Should the URL fail to validate completely, we should give a more useful error message.

The current message (example below) isn't very helpful.

Download of feed://example.org failed with code -1003.

Needs to be addressed upstream.

Attached patch attempts to replace any mention of feed:// with http:// in the source URL.

A check is done to ensure http:// is at the start of the source URL, if it is not - it gets added.

I think we should use a proven URL validator - what about valid_url() ?

I have nothing against replacing feed:// with http://. That seems like a reasonable guess.

Hi Will, Sorry it's taken so long to get back to you. I got the Twitter feeds to work using:
http://search.twitter.com/search.atom?q=microbicide
http://search.twitter.com/search.atom?q=microbicides

EATG Daily Digest feed still doesn't work. Most of the feeds I'm using are rolled in Google News from queries I've entered there. They seem to work pretty well. It's the single site ones, that don't always work, partly I imagine to do with how they do, or don't, implement RSS well.

Still not working:
http://avac.org/ht/a/getfeedaction/content/releases or http://www.avac.org/ht/a/getfeedaction/content/releases tried both, neither works
http://eatg.org/rss/feed/news or http://www.eatg.org/rss/feed/news

Here's the site, fyi: it's just a testsite on webenabled: http://www.preventhiv.dev3.webenabled.net/feeds/23 so, I haven't updated it to the most recent version of Managing News. That might help, too. I simply installed it, to show it to my co-workers, did no customization at all. We'll have a fresh installation in our new Drupal site, due to launch at end-October or soon after.

I don't know if Web Enabled installs curl for PHP, or not. Will check.

Thanks, Marilyn

The valid_url() function seemed like a waste of time in tests with random broken looking URLs, everything got through.

Attached patch is using the http_request_get_common_syndication() to validate the feed URL by catching the exception thrown. This also prevents the CURL error killing the site if a broken URL is provided for a feed and then the node saved.

@budda, I don't think you can assume that a given url is a feed. What if it's a CSV file, or an XML document that someone is parsing?

Also, if you were testing against urls that start with feed://, valid_url passes them on purpose.

This validation is only happening in the HTTPfetcher. It doesn't matter what data is in the file the URL points to. I wasn't proposing any file extension or MIME type validation.

No I wasn't testing against urls starting with feed:// -- but the code in the patch changed feed:// -> http:// before running a valid_url() anyway, so would make zero difference.

My mistake, I didn't realize that the http_request_get_common_syndication() call was only called if auto_detect_feeds was set.

This is looking great, except:

+    if (strpos($values['source'], 'http://') !== 0 ) {
+      $values['source'] = 'http://' . $values['source'];  
+    }

That's too aggressive. There can be valid feeds at https:// at least (what else are we missing?).

$values['source'] = str_replace('feed://', 'http://', $values['source']);

If we're at it, let's also convert the darn webcal:// scheme to http://

THe http:// prefix is a worst case. If a feed was on https:// then maybe the user should have pasted the full url in correctly to begin with?

Or are you thinking we should cycle through a known list of possibilities before validation failing? Such as ftp:// ? :-)

+    if (strpos($values['source'], 'http://') !== 0 ) {
+      $values['source'] = 'http://' . $values['source'];  
+    }

will convert a https://example.com to http://https://example.com.

I think he means the

if (strpos($values['source'], 'http://') !== 0 ) {

A valid url could start with https://.

Did our comments cross? The patch in #15 needs work because of #21.

I didn't mean to change that to needs review.

At the risk of being slightly off topic: a leading space in the feed url field turned out to be the cause of my "...failed with -1003" error.
Maybe this is the case for others out there too.

Since it's the user's responsibility to enter a correct url, maybe leading/trailing spaces can be caught in a form_validate and brought to the user's attention instead of accepting the value and later blocking on it. Would that be the right way to go?

#26: We can catch this reliably. I'd just add a trim() before saving.

I think the attached patch addresses the outstanding issues, the patch in #15 has been adjusted to use preg_match() and preg_replace() instead and incorporates the issue in #21.

Ignore the patch in #28 I grabbed the wrong one, sorry for the confusion.

Try this one instead. Tested with the following;

• webcal://rss.slashdot.org/Slashdot/slashdot
• feed://rss.slashdot.org/Slashdot/slashdot
• https://rss.slashdot.org/Slashdot/slashdot
• rss.slashdot.org/Slashdot/slashdot
• http://rss.slashdot.org/Slashdot/slashdot

The https one stays the same, but fails validation as that particular feed is not https enabled. All the others are adjusted to 'http://rss.slashdot.org/Slashdot/slashdot'

I don't think there's any need to change what the user has entered and it should just 'work'. And I don't think we should support non-protocol URLs at all.

1. We should add validation that at least runs through valid_url() to check for unsupported protocols.
2. Show a warning if the file did not fetch correctly.
3. Support the feed and webcal scheme directly inside http_request_get() just like drupal_http_request() does.
4. We currently don't return a different error if the URL has no scheme like drupal_http_request() does, so we should ensure that's added.

Gah, I had to make a copy of the valid_url() function that supports webcal:// until #295021: filter_var() with FILTER_VALIDATE_URL accepts malformed URLs and rejects not all valid URLs or #1191252: Add 'webcal' protocol support in valid_url() are fixed.

Committed #33 to Git:
http://drupalcode.org/project/feeds.git/commit/70f3c97
http://drupalcode.org/project/feeds.git/commit/2d553f7

And committed to 6.x-1.x!
http://drupalcode.org/project/feeds.git/commit/73aae88