indexer issue

A few ideas to try to sched some light:

1) Try invoking the XMLPipe command from the shell redirecting the output to a file. Then open the file to see how that line looks like. Maybe there's a PHP error, the file is interrupted, or something...

2) docid=198691 is the node ID. You could create a temporary main index with XMLPipe command where first/last node ID is that one. If that works, then it looks like the problem is somewhere else.

3) Try looking at admin/logs/watchdog filtering sphinxsearch reports. Maybe there's something there that it was missed before.

hmm... this seems to be \x1A which is certainly invalid for XML:

http://www.w3.org/International/questions/qa-controls

However, these are filtered by sphinxsearch_get_node_text(), located in sphinxsearch.module. But... this is only applied to build the "content" attribute of the XMLPipe document, it is not applied to the "subject" attribute. See sphinxsearch_xmlpipe_document() located in sphinxsearch.xmlpipe.inc. And this would be a nice bug :)

If you can take a look at the contents of the file you got when running the XMLPipe command from shell, probably the \x1A character is located within the <subject> element of the document. Could you please confirm?

If that's correct, the you could try the following:

1) Open the file sphinxsearch.xmlpipe.inc.
2) Find the function sphinxsearch_xmlpipe_document().
3) You'll seee something like this:

$output .= '<subject><![CDATA[['. check_plain($node->title) .']]></subject>' ."\n";

4) Before that line, add the follwing:

$node->title = preg_replace('#[\\x00-\\x08\\x0B\\x0C\\x0E-\\x1F]#S', ' ', $node->title);

Works?

If so, I'll post a patch and also create a new release for the D5 branch.

Great it worked. :) ...I'll try to pack a new release for the D5 branch, later today, if possible.

~

node_load() in sphinxsearch.common.inc is used to load nodes for the search results page, so there shouldn't be hundreds of nodes in memory. If you're concerned about this, please open a different issue.

As per indexer optimization... we already use node_load($nid, NULL, TRUE) in XMLPipe processing. Again, open a different issue if you wish to touch this subject. One suggestion from here: look at Drupal 6, in scripts directory, the scripy drupal.sh. It allows running Drupal from the CLI. This is something I discovered rencently, and I'll probably advice it's usage for the Drupal 6 version of this module. AFAICT, drupal.sh script is Drupal version agnostic, so you could get it from D6 and try it on your D5 installation.

Could you please generate a file with the result of redirecting the output of running the sphinxsearch_xmlpipe.php script with first/last nids set to the nid where it is failing?

I'll send you my email, so you can send me the file, and I try to see what happens on my system.

Hi,

As I sent you from email: "I've checked the XML stream you've sent me, and it also parses correctly on my system."

So I also suspect this is something related with some kind of processing limit. Here, I would suggest:

1) Check watchdog logs (filtering for sphinxsearch reports). XMLPipe generator for main index processing tries to record one report at start, and another at end of process with information about execution time, and memory used.

2) Compare the time it takes your index generation process with the PHP settings defined on your system, and or the .htaccess file stored on the sphinxsearch_scripts subdirectory.

3) Try increasing max_executions_time, memory_limit, mysql.connect_timeout, ...

4) Define one main index with as much nodes as possible. If it fails, try limiting the number of nodes using the first/last arguments of the xmlpipe script. This should help you find out how many nodes can be processed with you current PHP settings. So you can increase them, or otherwise you'll need to use more main indexes with less nodes per index.

Here's one possible method that could help you compute how many main indexes you need, and how many time it will take to reindex them:

1) Create a main index for (say) 1000 nodes and run the indexer.
2) Look at statistics for that process stored in admin/logs/watchdog. This should tell you node processed per second.

Now, if we got it took 50 nodes per second, then we can process 3,000 nodes per minute, or 180,000 nodes per hour. If we have 1.8million nodes, then we need 10 hours to index all nodes in one single index. If we create 10 indexes, then we need 1 hour per index, or maybe more if we rebuild more than index in parallel, depending on resources available on the server where Sphinx is installed (number of physical discs, raid5?, memory, cpus, etc.).

HTH+

Fix is included in 5.x-1.3 :)