My host times out the export process when it first starts. Nothing is printed to the error log or screen.

The batch urls run chronologically, so i knew that the current batch had to be 377, since the one before was 376 (as noted by using batch with a different module). So i inputted the new modified batch link & the export worked.

It would have been impossible for me to find this solution if I had not dealt with batch processes before. So to make things easier we should print an error message to the screen/ error log stating the batch url link to continue views data export process if it should fail/ time out. This error message should be printed 100% of the time.

Message/ error should say
"Click this link to continue the Views data export started at (insert timestamp) http://www.yourdomain.com/batch?id=377&op=do_nojs"
- the timestamp will help user discern between different exports when browsing through logs.
- replace 377 with your current batch #

Comments

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented
Title: Print batch url to log/error message » Index building/batch setup failed
Component: User interface » Code
Category: feature » bug

I think we'd like to keep it as simple as possible, and the batched export should just start in all cases.

I would be keen to know why your initial request timed out, and fix that issue though. The one weak point of the data export is the building of the index as the batch is set up. I wonder if you could provide more details:

What were you exporting?
What volume of data were you exporting?
Did the export work? Was all the data exported?
Can you share the view (export the view definition) that caused the issue?

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented
Status: Active » Postponed (maintainer needs more info)
TimelessDomain’s picture

TimelessDomain CreditAttribution: TimelessDomain commented

The host Rackspace Cloud Sites displays a "no suitable nodes error" (other hosts probably have a different message) instead of the batch url process. The url in the browser address bar is still the original url (not the batch url - this is where most people would get lost & why i requested the solution above).

More on the error:

"Basically, it is a generic error thrown by load balanced systems that occurs as a result of a script exceeding a maximum timeout value (not the PHP timeout value!) If a client connection does not receive a response from the server after approximately 30 to 60 seconds the load balancer will close the connection and the client will immediately receive the error message. In most cases, the script will continue to execute until it reaches completion, throws an error, or times out on the server, but the client will not see the page load as expected and will instead receive this error."

"Rackspace simply has a 30second cap on HTTP requests through the load balancer. All you need to do is run your larger scripts via ajax in the session and kick off a heartbeat (do a quick ping with a script that runs every 10-20 seconds) to fulfill the LB’s data request."

This problem does not stop the index from building, but it does stop you from being forwarded to the batch url.

More on the view:

Exporting: All Nodes and related fields

Export Volume: 140,000+ nodes

Results: All Data Was Exported (Using the manual type-in batch url solution noted above woked)

Views Definition Export: Download here - http://www.iseenews.com/isnexport.txt

I was able to get the export process work through correctly (non-stop) when using a simple view including:

- node title

- nid

- post date

When I added the following fields to the view the timeout (rackspace - no suitable nodes) error started appearing:

- statistics total views

- node path

- node body

- node published status

- All taxonomy terms: filtered to specific taxonomies

- embedded media field original url

- embedded media field duration

- cck link field

- cck text fields

- Vote Results vote up/down relationships & fields showing 1)total vote count

- Feed API Original URL

- + more

ACTUAL CSV EXPORT : A CSV of YouTube News Videos (+ a some news from digg & twitter #CBDQ design thinking hashtag)from last week to about 2 years ago(thanks to cck, feedapi, views & now views data export!). Download here: http://www.iseenews.com/allnews.csv (each video is tagged with source name)

THANK YOU Steven Jones for developing this module! It Rocks!

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented
Assigned: Unassigned » Steven Jones
Status: Postponed (maintainer needs more info) » Active

I'm not sure a huge amount can be done I'm afraid, I suspect that you've just hit another limit :(

I will have a think about what debugging measures you can take to give me more details so we can identify the issue more fully.

jamsilver’s picture

jamsilver CreditAttribution: jamsilver commented

I've recently noticed that mysql v5.1.6 and greater support triggering queries asynchronously by setting them up as one-time Events.

See: http://dev.mysql.com/tech-resources/articles/event-feature.html

I haven't looked into it in detail, but at a glance it looks like the perfect solution to this problem - if we fire off our query as an immediate event and forward the user onto batch processing straightaway.
We would add a new task to the batch that polls the mysql.events table to wait until the event has completed before moving onto the rendering task as normal.

Note, Drupal 7 officially supports mysql 5.0.15> (older for d6, d5..), so this would have to be a feature that degrades gracefully to the current behaviour if the version of mysql does not support events. Perhaps advise the user that if they upgrade their mysql they can take advantage of the feature.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented
Title: Index building/batch setup failed » Use MySQL events to build index
Assigned: Steven Jones » Unassigned
Category: bug » feature

Changing title.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented

The events concept puts quite high requirements on the server:

SET GLOBAL event_scheduler = 1;
To turn the feature off, do:

SET GLOBAL event_scheduler = 0;
As with all "SET GLOBAL" statements, you need the SUPER privilege to change the setting of the event_scheduler variable.

It is also possible to start the server with:

mysqld ... --event_scheduler=1
You can see what the current state of affairs is with

SHOW VARIABLES LIKE 'event_scheduler';
or
SELECT @@event_scheduler;

And there is a new permission for users that want to use events.

I'm not against implementing the events stuff, but we'll want to document it clearly so that people know that they could be missing out if they don't have events.

greggadsdon’s picture

greggadsdon CreditAttribution: greggadsdon at ComputerMinds commented
Issue summary: View changes
Status: Active » Closed (won't fix)

As Drupal 6 nears the end of its life and support period, the 6.x version of Views Data Export will not be supported either. To that end, this issue has been closed, and will not be resolved in the 6.x branch of Views Data Export.

If this issue is still relevant to 7.x branch of Views Data Export, then please re-open and move to a relevant 7.x version.