I've searched all over the place and either it's not out there or I missed it. Hopefully I missed it...

Anyway, I have my UberCart store all set up and running great, and now I want to start publishing to shopping feeds. Is there a module that will easily export my products to google base and/or any of the other shopping feed sites?

Thanks in advance and sorry if the answer is obvious and I missed it...

Comments

scarr’s picture

Hi rs7272 -

There is a google base module; you can find it here: http://drupal.org/project/googlebase and another version here: http://www.ubercart.org/contrib/11593

They are not very usable at this point (no offense to the contributors! I'd love to see one of these fleshed out and built up.)

What I've done is used the views bonus pack (http://drupal.org/project/views_bonus) to export my feeds in various formats. The CSV export does a fairly good job, but the RSS and XML don't allow me to insert the namespace declarations needed to produce good feeds at this time (and I'm sure some of the problem is my own lack of expertise in code and theming).

So far I've successfully submitted feeds to Pricegrabber, Nextag, and Google Base. I'm working on Shopping and Shopzilla.

The main issue I've had is that the description field, which is usually mapped from the body of the product node, is full of things that aren't good for the feed, like commas, character-encodings, html tags (which can be stripped) and more. So the file sometimes fails, and for a few of these feeds, I've just had to use the title as description until I find a solution.

I'd be happy to show you how I've structured things, and if you have any ideas for some of the problems I'm facing, I'm all ears.

dzieyzone’s picture

sunward’s picture

dzieyzone , nice work (not tried yet) but how do you customize the fields for made in, isbn, and product type?

danharper’s picture

I've tried this but it doesn't seem to load any variables into the feed, possibly to do with views 3.

any helpd would be appreciated.

Cheers Dan

danharper’s picture

I needed the nid as a field in the view.

Great Post

Thanks Dan

bocaj’s picture

@scarr Do you mind sharing how you setup views bonus pack to export your feed into usable data for Google Base? I am assuming that there is some editing to the file that is exported as well before submitting to Google? I'd be more than willing to try to help you figure out your issues (if you haven't figured them out yet) once I have an idea of what's going on. It sounds like an issue that I may run into as well with our website.

Thanks!

scarr’s picture

Hi bocaj and everyone,

Sorry for the long time in between posts - work took me to other issues and then back to feeds. Good news is that I've figured it out, at least for the most part (we are still affecting some changes, and I'll post when I have them).

Because of the number of products on this particular site (+1400), I needed a solution which required no editing after exporting. I have google pulling the feed directly from the site (you can see it at http://www.unique-sports.com/exports/google.txt)

For google base, we created a view using the csv file type. I changed the file name to end in .txt and changed the comma delimiter with a tab (I tabbed in notepad++ and copy/pasted it into that field).

I arranged the fields based on what google merchant center fields I wanted to include. Remember to change the label to the header google is looking for.

At this point, everything was nearly fine, except the feed kept breaking. The issue was encoding, as paragraph tags from the body (as well as copyright symbols, etc.) broke the tab delimitation. To solve this, my developers created a custom php snippet (which we are finalizing; after we finish it up, I will post for everyone).

The feed uploaded successfully, except for a few products that we have in there as duplicates and need to clean up. Google is downloading the feed each night and we are showing well in google products and such.

I've thought about creating a longer version of this post, as a how-to to create great feeds - would that be useful?

Thanks

Steve

sunward’s picture

"I've thought about creating a longer version of this post, as a how-to to create great feeds - would that be useful?"

yes please

BTW, the txt file is over 2mb in size. Is this normal?

scarr’s picture

Hi Sunward,

I'll work on a full write-up. In the meantime, I'll be happy to answer questions. As for the file size, that is unfortunately due to the fact that even when I set a character limit on the body/description, for some reason the feed seems to be pulling the entire thing. The client has written some pretty expansive descriptions, and that plus the thousands of products have created a pretty big file. I'm having my developer look at that to make sure nothing odd is happening (which it probably is, haha) but google seems not to have a problem.

One thing I would like, and I'm not sure how to accomplish, is for drupal to create this feed and place a real file on the server, rather than have it generate the file when it is called by google. That would prevent some speed issues when google pulls the feed (which I have set for 2am, so no big deal).

Thanks

theohawse’s picture

A not so quick solution to the download time could be to break your feeds into groups say 250 products , each and have one final feed group that holds more than 250. Then have google import one per hour or so. This would only work if google merchant accepts more than one feed. There could also be caching benefits to this. Also you could split feeds by taxonomy.

scarr’s picture

Google does allow more than one feed, though the other major comparison shopping feed companies do not - this will have to be addressed at some point, though it's probably not terribly pressing. I believe that the rebuilt xml sitemap module now stores a file on the server built after cron runs, though I'm not 100% on that - it's possible that we could convince views bonus pack to mirror this if its worth pursuing.

asad.hasan’s picture

function google_feed_perm() {
	return array('export content');	
}

function google_feed_forms() {
	$forms = array();
	$forms['google_feed_export_form'] = array(
		'callback' => 'google_feed_export_form_builder',
	);
	
	return $forms;
}

function google_feed_export_form_builder(&$form_state) {
	$form = array();
	
	$form['export'] = array(
		'#type' => 'submit',
		'#value' => t('Export Google Feed'),
		'#submit' => array('_google_feed_process')
	);
	
	return $form;
}

function _google_feed_process($form, &$form_state) {
	$dir = file_directory_path().'/google_feeds';
	if(file_check_directory($dir, FILE_CREATE_DIRECTORY, 'export')) {
		$output_location = file_directory_path().'/google_feeds/google_feed.xml';
		
		global $base_url;
		$header = '<?xml version="1.0" encoding="UTF-8" ?>'."\n";
		$header .= '<rss version ="2.0" xmlns:g="http://base.google.com/ns/1.0">'."\n";
		$header .= '<channel>'."\n";
		$header .= '<title>Product Feed - '.variable_get('site_name', "").'</title>'."\n";
		$header .= '<description>Concise data feed of products</description>'."\n";
		$header .= '<link>'.$base_url.base_path().'</link>'."\n";

		$batch = array(
			'title' => t('Exporting Google Feed'),
			'operations' => array(array('_google_feed_batch_iteration', array($output_location))),
			'finished' => '_google_feed_batch_finished'
		);
	
		batch_set($batch);
		
		$fp = fopen($output_location, 'w');
		fwrite($fp, $header);
		fclose($fp);
	}
}

function _google_feed_batch_iteration($output_location, &$context) {
	$fp = fopen($output_location, 'a');

	if(empty($context['sandbox'])) {
		$context['sandbox']['progress'] = 0;
		$context['sandbox']['current_node'] = db_result(db_query("SELECT MAX(n.nid) FROM {node} n WHERE n.type IN ('cellphones', 'batteries', 'chargers', 'headsets', 'pouches', 'sim_cards', 'screen_protectors', 'memory', 'data_cables', 'phone_holders', 'bluetooth', 'charms', 'misc', 'antennas_and_amplifiers');")) + 1;
		
		//Get just instock items. Have a configurable stock vocab, or cck field
		//$context['sandbox']['max'] = db_result(db_query("SELECT COUNT(DISTINCT n.nid) FROM {node} n INNER JOIN {content_field_stock_type} s ON n.vid = s.vid WHERE n.type IN ('cellphones', 'batteries', 'chargers', 'headsets', 'pouches', 'sim_cards', 'screen_protectors', 'memory', 'data_cables', 'phone_holders', 'bluetooth', 'charms', 'misc', 'antennas_and_amplifiers') AND s.field_stock_type_value IN ('instock', 'backordered');"));
		//$context['sandbox']['max'] = 30;
		
		//Get all stock types
		$context['sandbox']['max'] = db_result(db_query("SELECT COUNT(DISTINCT n.nid) FROM {node} n WHERE n.type IN ('cellphones', 'batteries', 'chargers', 'headsets', 'pouches', 'sim_cards', 'screen_protectors', 'memory', 'data_cables', 'phone_holders', 'bluetooth', 'charms', 'misc', 'antennas_and_amplifiers');"));
	}
	
	$limit = 2; //number of dumps per batch iteration
	//$result = db_query_range("SELECT n.nid FROM {node} n INNER JOIN {content_field_stock_type} s ON n.vid = s.vid WHERE n.nid > %d AND n.type IN ('cellphones', 'batteries', 'chargers', 'headsets', 'pouches', 'sim_cards', 'screen_protectors', 'memory', 'data_cables', 'phone_holders', 'bluetooth', 'charms', 'misc', 'antennas_and_amplifiers') ORDER BY n.nid ASC", $context['sandbox']['current_node'], 0, $limit);
	//Get newer ones first
	$result = db_query_range("SELECT n.nid FROM {node} n WHERE n.nid < %d AND n.type IN ('cellphones', 'batteries', 'chargers', 'headsets', 'pouches', 'sim_cards', 'screen_protectors', 'memory', 'data_cables', 'phone_holders', 'bluetooth', 'charms', 'misc', 'antennas_and_amplifiers') ORDER BY n.nid DESC", $context['sandbox']['current_node'], 0, $limit);

	while ($row = db_fetch_array($result)) {
		$node = node_load($row['nid'], NULL, TRUE);
		$context['results'][] = $node->nid;
		$context['sandbox']['progress']++;
		$context['sandbox']['current_node'] = $node->nid;
		$context['message'] = 'Currently Exporting: '.$node->title;
		fwrite($fp, _google_feed_item($node));
	}
	
	if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
		$context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
	}
	fclose($fp);
}

function _google_feed_batch_finished($success, $results, $operations) {
	global $base_url;
	
	$output_location = file_directory_path().'/google_feeds/google_feed.xml';
	$closure = '</channel>'."\n";
	$closure .= '</rss>'."\n";
	$fp = fopen($output_location, 'a');
	fwrite($fp, $closure);
	fclose($fp);
	
	if ($success) {
		$merchant_details = "\nPlease point your Google Merchant account to ".$base_url.'/'.$output_location.", if you haven't already done so.";
		$message = format_plural(count($results), 'One product was exported.'.$merchant_details, '@count products were exported.'.$merchant_details);
  	} else {
    	$message = t('An error occured while exporting, please contact D.A. Designers if problem presists.');
	}
  	drupal_set_message($message);
}

function google_feed_menu() {
	$urls = array();
	
	$urls['admin/content/google_feed_export'] = array(
		'title' => 'Export Google Feed',
		'description' => 'Export all published products to a file in Google Merchant format.',
		'access callback' => 'user_access',
		'access arguments' => array('export content'),
		'page callback' => 'drupal_get_form',
		'page arguments' => array('google_feed_export_form'),
	);
	
	return $urls;
}

function _google_feed_item($node) {
	$title = check_plain($node->title);
	$content = '';
	if(!empty($title)) {
		//Make sure title isn't empty
		$content .= '<item>'."\n";
		$content .= '<g:id>'.$node->nid.'</g:id>'."\n";
		$content .= '<title>'.$title.'</title>'."\n";
		$content .= '<link>'.url('node/'.$node->nid, array('absolute' => TRUE)).'</link>'."\n";
		$content .= '<g:price>'.$node->sell_price.' USD</g:price>'."\n";
		$content .= '<description>'.htmlspecialchars(html_entity_decode(preg_replace('/(&nbsp);*/', ' ', substr(filter_xss($node->teaser, array()), 0, 10000)))).'</description>'."\n";
		$content .= _google_feed_get_term($node, 29, 'condition', array("Brand New" => "new", "Refurbished" => "refurbished"), "new");
		$content .= _google_feed_get_term($node, 22, 'brand');
		$content .= '<g:mpn>'.check_plain($node->model).'</g:mpn>'."\n";
		$content .= '<g:image_link>'.url($node->field_image_cache[0]["filepath"], array('absolute' => TRUE)).'</g:image_link>'."\n";
		$content .= '<g:product_type>'.check_plain($node->type).'</g:product_type>'."\n";
		$content .= _google_feed_quantity($node->field_stock_type[0]["value"]);
		$content .= '<g:availability>'._google_feed_convert_strings($node->field_stock_type[0]["value"], array("instock" => "in stock", "backordered" => "limited availability", "outofstock" => "out of stock", "discontinued" => "out of stock", "automatic" => "in stock"), "in stock").'</g:availability>'."\n";
		$content .= _google_feed_get_term($node, 51, 'feature');
		$content .= '<g:online_only>n</g:online_only>'."\n";
		$content .= _google_feed_get_term($node, 26, 'manufacturer')._google_feed_get_term($node, 19, 'manufacturer')._google_feed_get_term($node, 33, 'manufacturer')._google_feed_get_term($node, 49, 'manufacturer')._google_feed_get_term($node, 46, 'manufacturer')._google_feed_get_term($node, 41, 'manufacturer')._google_feed_get_term($node, 35, 'manufacturer')._google_feed_get_term($node, 17, 'manufacturer')._google_feed_get_term($node, 38, 'manufacturer')._google_feed_get_term($node, 34, 'manufacturer')._google_feed_get_term($node, 39, 'manufacturer');
		$content .= '<g:featured_product>'.$node->promote.'</g:featured_product>'."\n";
		$content .= '</item>'."\n";
	}
	
	return $content;
}

function _google_feed_convert_strings($string, $mapper, $default = '') {
	foreach($mapper as $key => $val) {
		if($key == $string) {
			return $val;
		}
	}
	return $default;
}

function _google_feed_quantity($stock_value) {
	if($stock_value == "outofstock" || $stock_value == "discontinued") {
		return '<g:quantity>0</g:quantity>'."\n";
	} else {
	 	return '';	
	}
}

function _google_feed_get_term($node, $vid, $tag, $mapper = array(), $default = '') {
	$terms = taxonomy_node_get_terms_by_vocabulary($node, $vid);
	foreach($terms as $term) {
		if(empty($mapper)) {
			return '<g:'.$tag.'>'.check_plain($term->name).'</g:'.$tag.'>'."\n";
		} else {
			return '<g:'.$tag.'>'.check_plain(_google_feed_convert_strings($term->name, $mapper)).'</g:'.$tag.'>'."\n";
		}
	}

	if(empty($default)) {
		return '';
	} else {
		return '<g:'.$tag.'>'.check_plain($default).'</g:'.$tag.'>'."\n";
	}
}

Manufacturer was a taxonomy term based on different product types. And teh product types are hard coded, i will make it more dynamic and configurable over time. For those that can modify code and use it feel free, or help me make it configurable.

Thanks.